2024-03-29T11:23:47.614 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:23:47.614 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --verbose --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732/orig.config.yaml 2024-03-29T11:23:47.648 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:23:47.732 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622732' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5bf0bJYXI1z2YAYj2AirhKKyd1vhJ3GYcn3DfjFSYffBbXRw3uQpBRHkGxaMS3Jk2YN+vvtOZrRkqzX6g7kVY= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCXqkS0wvhnvppYL6bO+4o1t1ZrPFTGqFlyuLbR513BpWS3w1HfG9bfqSonOx7xNWAp6+8jBREvVeGStEZIeuuo= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:23:47.732 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:23:47.733 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:23:47.733 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:23:47.735 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:23:47.760 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:23:47.760 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:23:47.760 INFO:teuthology.packaging:ref: None 2024-03-29T11:23:47.760 INFO:teuthology.packaging:tag: None 2024-03-29T11:23:47.760 INFO:teuthology.packaging:branch: main 2024-03-29T11:23:47.760 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:23:47.760 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:23:48.012 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:23:48.013 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:23:48.020 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:23:48.020 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:23:48.029 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:23:48.042 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:23:48.049 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:23:48.073 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:18:45.407652', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5bf0bJYXI1z2YAYj2AirhKKyd1vhJ3GYcn3DfjFSYffBbXRw3uQpBRHkGxaMS3Jk2YN+vvtOZrRkqzX6g7kVY='} 2024-03-29T11:23:48.098 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:18:45.409497', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCXqkS0wvhnvppYL6bO+4o1t1ZrPFTGqFlyuLbR513BpWS3w1HfG9bfqSonOx7xNWAp6+8jBREvVeGStEZIeuuo='} 2024-03-29T11:23:48.098 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:23:48.104 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:23:48.104 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:23:48.104 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:23:48.178 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1d7b6330d0>, signals=[15]) 2024-03-29T11:23:48.178 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:23:48.185 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:23:48.185 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:23:48.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:23:48.275 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:23:48.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:23:48.364 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:23:48.370 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2024-03-29T11:23:48.387 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2024-03-29T11:23:48.388 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2024-03-29T11:23:48.443 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="9" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="9" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:23:48.444 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:23:48.445 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2024-03-29T11:23:48.467 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-03-29T11:23:48.485 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-03-29T11:23:48.485 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-03-29T11:23:48.539 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-03-29T11:23:48.539 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:23:48.540 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:23:48.541 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-03-29T11:23:48.567 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:23:48.576 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:23:48.581 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:23:48.581 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:23:48.584 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:23:48.600 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:23:48.606 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:23:48.606 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:23:48.640 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:23:48.672 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:23:48.723 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:23:48.737 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:23:48.737 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:23:48.738 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:23:48.738 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:23:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:23:48.754 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-03-29T11:23:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:23:48.755 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-03-29T11:23:48.810 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:23:48.810 INFO:teuthology.task.kernel:Running kernel on smithi100: 5.14.0-361.el9.x86_64 2024-03-29T11:23:48.810 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum install -y kernel 2024-03-29T11:23:48.811 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:23:48.812 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-361.el9.x86_64 2024-03-29T11:23:48.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2024-03-29T11:23:50.955 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:23:51.001 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:23:52.593 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-29T11:23:52.598 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-29T11:23:53.608 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T11:23:53.863 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 8.4 MB/s | 8.0 MB 00:00 2024-03-29T11:23:55.601 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-29T11:23:56.506 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-29T11:24:00.682 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.1 MB 00:00 2024-03-29T11:24:01.043 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.1 MB 00:01 2024-03-29T11:24:02.141 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-03-29T11:24:02.904 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 16 kB 00:00 2024-03-29T11:24:03.913 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-29T11:24:08.863 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-29T11:24:10.983 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 2.4 MB/s | 21 MB 00:08 2024-03-29T11:24:11.119 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:24:11.119 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:24:11.208 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:24:11.212 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:24:11.212 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:24:11.212 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:24:11.212 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:24:11.213 INFO:teuthology.orchestra.run.smithi100.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:24:11.213 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-29T11:24:11.213 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:24:11.213 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:24:11.213 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:24:11.213 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:24:11.213 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:24:11.213 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:24:11.213 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-03-29T11:24:11.213 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:24:11.214 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 76 M 2024-03-29T11:24:11.214 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 123 M 2024-03-29T11:24:11.214 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:24:11.372 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:24:11.839 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-03-29T11:24:12.439 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T11:24:12.748 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T11:24:12.751 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:24:12.752 INFO:teuthology.orchestra.run.smithi100.stdout:Total 50 MB/s | 76 MB 00:01 2024-03-29T11:24:12.752 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:24:13.106 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:24:13.106 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:24:13.965 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:24:13.965 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:24:16.000 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T11:24:16.278 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:24:17.103 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:24:17.191 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:24:18.299 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:24:18.299 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:24:18.408 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:24:18.413 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:24:18.413 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:24:18.413 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:24:18.413 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:24:18.413 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 76 M 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 123 M 2024-03-29T11:24:18.414 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:24:18.576 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:24:19.002 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 26 MB/s | 15 MB 00:00 2024-03-29T11:24:19.635 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T11:24:19.939 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:24:20.043 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T11:24:20.047 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:24:20.047 INFO:teuthology.orchestra.run.smithi191.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-29T11:24:20.047 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:24:20.078 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:24:20.418 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:24:20.418 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:24:21.268 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:24:21.268 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:24:23.583 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:24:24.187 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:24:24.270 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:24:24.760 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:24:24.891 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:27.084 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:24:27.224 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:24:29.618 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:31.941 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:24:32.056 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:36.779 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:58.829 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:24:58.829 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:24:58.872 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:24:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:24:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:24:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:24:59.698 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:24:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:24:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:24:59.699 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:25:00.012 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:25:00.165 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:25:00.166 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:25:00.166 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:25:00.166 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:25:00.166 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:25:00.166 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:25:00.166 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:25:00.167 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:25:00.246 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:25:00.246 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:25:00.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2024-03-29T11:25:00.283 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.06 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 68.el9 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Group : Unspecified 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 8264254 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:25:00.284 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : builder@centos.org 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:25:00.285 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:25:00.286 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:25:00.286 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:25:00.286 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:25:00.935 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2024-03-29T11:25:02.294 INFO:teuthology.orchestra.run.smithi100.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:25:02.328 INFO:teuthology.orchestra.run.smithi100.stderr:done 2024-03-29T11:25:02.330 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-03-29T11:25:02.346 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.UzlVCgu9hH 2024-03-29T11:25:02.346 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UzlVCgu9hH 2024-03-29T11:25:02.421 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.UzlVCgu9hH 2024-03-29T11:25:02.561 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.UzlVCgu9hH is 6KB 2024-03-29T11:25:02.609 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.UzlVCgu9hH 2024-03-29T11:25:02.623 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:25:02.687 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:25:02.687 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:25:02.687 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:25:02.687 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:25:02.689 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:25:02.846 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2024-03-29T11:25:06.524 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:25:06.524 INFO:teuthology.orchestra.run.smithi191.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:25:06.524 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:25:06.567 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:25:06.966 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:25:06.966 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:25:06.966 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:25:06.967 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:25:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:25:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:25:07.368 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:25:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:25:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:25:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:25:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:25:07.368 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:25:07.369 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:25:07.714 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:25:07.818 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:25:07.818 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:25:07.818 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:25:07.818 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:25:07.818 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:25:07.818 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:25:07.819 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:25:07.819 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:25:07.899 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:25:07.899 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:25:07.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2024-03-29T11:25:07.937 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2024-03-29T11:25:07.937 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 68.el9 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8264254 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:25:07.938 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:25:07.939 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:25:07.940 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:25:07.940 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:25:07.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:25:08.578 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-03-29T11:25:09.999 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:25:10.029 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-03-29T11:25:10.031 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-03-29T11:25:10.046 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.LGvPRO4squ 2024-03-29T11:25:10.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LGvPRO4squ 2024-03-29T11:25:10.116 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.LGvPRO4squ 2024-03-29T11:25:10.255 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.LGvPRO4squ is 6KB 2024-03-29T11:25:10.306 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.LGvPRO4squ 2024-03-29T11:25:10.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:25:10.391 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:25:10.391 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:25:10.391 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:25:10.391 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:25:10.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:25:10.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-03-29T11:25:32.867 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:25:32.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-29T11:25:32.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:40.559 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:25:40.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-29T11:25:40.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:26:32.925 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:26:40.567 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:26:41.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-29T11:26:41.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:26:42.301 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-03-29T11:26:42.814 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-29T11:26:42.814 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:26:42.814 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-03-29T11:26:42.828 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:26:42.829 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:26:42.829 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:26:42.829 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:26:43.829 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:26:43.830 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:26:43.884 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2024-03-29T11:26:43.920 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:49.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-29T11:26:49.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:26:53.063 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-29T11:26:53.516 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-29T11:26:53.516 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:26:53.516 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-03-29T11:26:53.530 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:26:53.531 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:26:53.531 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:26:53.531 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:26:54.532 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:26:54.532 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:26:54.597 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-03-29T11:26:54.632 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:54.633 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:26:54.640 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:26:54.640 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:54.643 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:54.659 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:26:54.666 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:26:54.672 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:26:54.673 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:54.701 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:54.739 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:26:54.745 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:26:54.745 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:54.776 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:54.808 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:54.812 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:54.823 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:54.826 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:54.827 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:26:54.835 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:26:54.835 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:54.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:54.897 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:26:54.910 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:26:54.911 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:54.940 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:54.956 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:55.041 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:55.093 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:55.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:55.220 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:26:55.220 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:55.291 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:55.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:55.381 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:55.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:55.504 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:26:55.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:55.573 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2024-03-29T11:26:55.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-03-29T11:26:55.631 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:55.663 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:56.130 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:26:56.138 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:26:56.138 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:26:56.149 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:26:56.159 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2024-03-29T11:26:56.237 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2024-03-29T11:26:56.239 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-03-29T11:26:56.283 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-03-29T11:26:56.285 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:26:56.286 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2024-03-29T11:26:56.316 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2024-03-29T11:26:56.317 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-03-29T11:26:56.343 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-03-29T11:26:56.343 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi100.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:26:56.343 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:56.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:56.415 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:26:56.416 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:56.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:56.493 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:26:56.503 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:26:56.522 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:26:56.524 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi100.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:31:13.477 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi100.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-03-29T11:31:13.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-29T11:31:13.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:31:13.570 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-03-29T11:31:13.645 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-29T11:31:13.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-29T11:31:13.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:31:13.737 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-29T11:31:13.822 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-29T11:31:13.822 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:31:13.833 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:31:13.834 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:31:13.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:31:13.837 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:31:13.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:31:13.866 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:31:13.883 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:31:13.893 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:31:13.909 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:31:13.911 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: ntpd: command not found 2024-03-29T11:31:13.925 INFO:teuthology.orchestra.run.smithi100.stdout:506 Cannot talk to daemon 2024-03-29T11:31:13.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-03-29T11:31:13.941 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:31:13.952 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-03-29T11:31:13.958 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:31:13.969 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:31:13.985 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:31:14.010 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:31:14.013 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:31:14.013 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-03-29T11:31:14.014 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:14.014 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:14.014 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:14.014 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:14.039 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:31:14.043 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:31:14.043 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-29T11:31:14.043 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:14.043 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:14.043 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:14.044 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:31:14.044 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:31:14.053 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:31:14.053 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:31:14.053 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:31:14.053 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:31:14.059 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:31:14.059 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:31:14.059 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:31:14.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:31:14.061 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:31:14.062 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:31:14.201 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:31:14.201 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:31:14.205 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:31:14.205 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:31:14.487 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:31:14.488 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:31:14.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:31:14.515 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:31:14.516 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:31:14.520 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:31:14.520 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:31:14.520 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:31:14.548 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:31:14.548 DEBUG:teuthology.orchestra.run.smithi100:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:31:14.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:31:14.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:31:14.662 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:31:14.696 DEBUG:teuthology.orchestra.run.smithi100:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:31:14.729 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-03-29T11:31:14.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-03-29T11:31:14.762 INFO:teuthology.orchestra.run.smithi100.stdout:check_obsoletes = 1 2024-03-29T11:31:14.764 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum clean all 2024-03-29T11:31:14.962 INFO:teuthology.orchestra.run.smithi191.stdout:74 files removed 2024-03-29T11:31:14.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw 2024-03-29T11:31:15.000 INFO:teuthology.orchestra.run.smithi100.stdout:74 files removed 2024-03-29T11:31:15.026 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-radosgw 2024-03-29T11:31:16.149 INFO:teuthology.orchestra.run.smithi100.stdout:ceph packages for x86_64 99 kB/s | 82 kB 00:00 2024-03-29T11:31:16.206 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 89 kB/s | 82 kB 00:00 2024-03-29T11:31:17.025 INFO:teuthology.orchestra.run.smithi100.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-29T11:31:17.066 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-29T11:31:17.858 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-29T11:31:17.959 INFO:teuthology.orchestra.run.smithi100.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2024-03-29T11:31:18.319 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:31:18.469 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:31:19.703 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-29T11:31:19.870 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-29T11:31:24.119 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:04 2024-03-29T11:31:25.135 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:04 2024-03-29T11:31:26.204 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-29T11:31:27.281 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-29T11:31:30.522 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 6.1 MB 00:00 2024-03-29T11:31:31.939 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.1 MB 00:01 2024-03-29T11:31:32.149 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-29T11:31:33.604 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-29T11:31:36.959 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 21 MB 00:03 2024-03-29T11:31:41.931 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-29T11:31:44.512 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:31:44.518 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:31:44.518 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:31:44.518 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:31:44.518 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:31:44.518 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrading: 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:44.519 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:44.520 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:44.520 INFO:teuthology.orchestra.run.smithi100.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:44.520 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:44.520 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:44.520 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:44.520 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:44.520 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:44.520 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:44.520 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:44.520 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:44.521 INFO:teuthology.orchestra.run.smithi100.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:44.522 INFO:teuthology.orchestra.run.smithi100.stdout:Installing weak dependencies: 2024-03-29T11:31:44.522 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:44.522 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:31:44.522 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:31:44.522 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:31:44.522 INFO:teuthology.orchestra.run.smithi100.stdout:Install 30 Packages 2024-03-29T11:31:44.522 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrade 2 Packages 2024-03-29T11:31:44.522 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:31:44.523 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 58 M 2024-03-29T11:31:44.523 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:31:49.623 INFO:teuthology.orchestra.run.smithi100.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 5.3 MB 00:04 2024-03-29T11:31:49.748 INFO:teuthology.orchestra.run.smithi100.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 202 kB/s | 25 kB 00:00 2024-03-29T11:31:50.473 INFO:teuthology.orchestra.run.smithi100.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 742 kB 00:00 2024-03-29T11:31:50.932 INFO:teuthology.orchestra.run.smithi100.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.1 MB/s | 498 kB 00:00 2024-03-29T11:31:52.640 INFO:teuthology.orchestra.run.smithi100.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 9.7 MB 00:08 2024-03-29T11:31:52.740 INFO:teuthology.orchestra.run.smithi100.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-29T11:31:52.857 INFO:teuthology.orchestra.run.smithi100.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.1 MB/s | 131 kB 00:00 2024-03-29T11:31:53.049 INFO:teuthology.orchestra.run.smithi100.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 845 kB/s | 162 kB 00:00 2024-03-29T11:31:53.342 INFO:teuthology.orchestra.run.smithi100.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.1 MB/s | 320 kB 00:00 2024-03-29T11:31:53.633 INFO:teuthology.orchestra.run.smithi100.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.0 MB/s | 298 kB 00:00 2024-03-29T11:31:53.734 INFO:teuthology.orchestra.run.smithi100.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 993 kB/s | 99 kB 00:00 2024-03-29T11:31:53.842 INFO:teuthology.orchestra.run.smithi100.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-29T11:31:53.884 INFO:teuthology.orchestra.run.smithi100.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T11:31:53.909 INFO:teuthology.orchestra.run.smithi100.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-29T11:31:53.959 INFO:teuthology.orchestra.run.smithi100.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-29T11:31:54.002 INFO:teuthology.orchestra.run.smithi100.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T11:31:54.027 INFO:teuthology.orchestra.run.smithi100.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-29T11:31:54.060 INFO:teuthology.orchestra.run.smithi100.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2024-03-29T11:31:54.086 INFO:teuthology.orchestra.run.smithi100.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.4 MB/s | 35 kB 00:00 2024-03-29T11:31:54.111 INFO:teuthology.orchestra.run.smithi100.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-29T11:31:54.486 INFO:teuthology.orchestra.run.smithi100.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 824 kB/s | 309 kB 00:00 2024-03-29T11:31:54.795 INFO:teuthology.orchestra.run.smithi100.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-29T11:31:54.854 INFO:teuthology.orchestra.run.smithi100.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-03-29T11:31:54.913 INFO:teuthology.orchestra.run.smithi100.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 835 kB/s | 49 kB 00:00 2024-03-29T11:31:54.979 INFO:teuthology.orchestra.run.smithi100.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T11:31:55.063 INFO:teuthology.orchestra.run.smithi100.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-29T11:31:55.139 INFO:teuthology.orchestra.run.smithi100.stdout:(27/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 5.1 MB 00:04 2024-03-29T11:31:55.163 INFO:teuthology.orchestra.run.smithi100.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-29T11:31:55.639 INFO:teuthology.orchestra.run.smithi100.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 3.2 MB/s | 1.6 MB 00:00 2024-03-29T11:31:57.731 INFO:teuthology.orchestra.run.smithi100.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 3.1 MB 00:02 2024-03-29T11:31:58.872 INFO:teuthology.orchestra.run.smithi100.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 931 kB/s | 3.4 MB 00:03 2024-03-29T11:31:59.681 INFO:teuthology.orchestra.run.smithi100.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 20 MB 00:15 2024-03-29T11:31:59.686 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:59.686 INFO:teuthology.orchestra.run.smithi100.stdout:Total 3.8 MB/s | 58 MB 00:15 2024-03-29T11:31:59.687 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:31:59.724 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:31:59.725 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:32:00.188 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:32:00.189 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:32:01.631 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:32:01.863 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:32:01.952 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:32:01.977 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:32:02.002 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:32:02.018 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:32:02.219 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:32:02.273 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:32:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:32:02.662 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:32:02.762 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:32:02.849 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:32:02.881 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:32:02.926 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:32:02.955 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:32:02.998 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:32:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:32:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:32:03.154 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:32:03.189 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:32:03.296 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:32:03.340 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:32:03.381 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:32:03.422 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:32:03.451 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:32:03.484 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:32:03.821 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:32:03.855 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:32:03.945 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:32:03.977 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:32:04.020 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:32:04.505 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:32:04.534 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:32:04.606 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:32:04.680 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:32:04.710 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:32:06.915 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:32:06.965 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:32:07.515 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:32:07.551 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:32:07.623 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:32:07.624 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:32:07.624 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:32:07.624 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:07.661 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:16.836 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:16.836 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /sys 2024-03-29T11:32:16.836 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /proc 2024-03-29T11:32:16.837 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /mnt 2024-03-29T11:32:16.837 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /var/tmp 2024-03-29T11:32:16.837 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /home 2024-03-29T11:32:16.837 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /root 2024-03-29T11:32:16.837 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /tmp 2024-03-29T11:32:16.837 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:17.721 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:17.745 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:17.745 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:17.745 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:32:17.746 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:17.746 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:17.746 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:17.747 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:17.832 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:17.832 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:19.025 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:19.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:32:19.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:32:19.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:32:19.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:32:19.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:32:19.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:32:19.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:32:19.026 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:32:19.027 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:32:19.028 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:32:20.112 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout:Upgraded: 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:32:20.113 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.114 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:20.115 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:32:20.460 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-test 2024-03-29T11:32:21.011 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:32:21.168 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:32:21.169 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:32:21.169 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:32:21.169 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:32:21.169 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:32:21.169 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout:Install 6 Packages 2024-03-29T11:32:21.170 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:21.171 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 48 M 2024-03-29T11:32:21.171 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 204 M 2024-03-29T11:32:21.171 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:32:21.480 INFO:teuthology.orchestra.run.smithi100.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-29T11:32:21.580 INFO:teuthology.orchestra.run.smithi100.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.0 MB/s | 300 kB 00:00 2024-03-29T11:32:21.613 INFO:teuthology.orchestra.run.smithi100.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:32:21.748 INFO:teuthology.orchestra.run.smithi100.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:32:21.914 INFO:teuthology.orchestra.run.smithi100.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 259 kB/s | 163 kB 00:00 2024-03-29T11:32:45.329 INFO:teuthology.orchestra.run.smithi100.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 47 MB 00:24 2024-03-29T11:32:45.331 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:45.332 INFO:teuthology.orchestra.run.smithi100.stdout:Total 2.0 MB/s | 48 MB 00:24 2024-03-29T11:32:45.345 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:32:45.366 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:32:45.366 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:32:45.741 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:32:45.741 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:32:46.317 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:32:46.396 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:46.431 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:46.455 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:46.495 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:46.523 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:51.023 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:51.114 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:53.448 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:53.449 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:53.449 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:53.449 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:53.449 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:53.449 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:53.887 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:32:54.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph 2024-03-29T11:32:54.757 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:13 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:32:54.917 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:32:54.919 INFO:teuthology.orchestra.run.smithi100.stdout:========================================================================================= 2024-03-29T11:32:54.919 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout:========================================================================================= 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:54.920 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:54.921 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout:========================================================================================= 2024-03-29T11:32:54.922 INFO:teuthology.orchestra.run.smithi100.stdout:Install 36 Packages 2024-03-29T11:32:54.923 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:32:54.923 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 30 M 2024-03-29T11:32:54.923 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 107 M 2024-03-29T11:32:54.923 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:32:56.008 INFO:teuthology.orchestra.run.smithi100.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-29T11:32:56.783 INFO:teuthology.orchestra.run.smithi100.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 1.5 MB 00:01 2024-03-29T11:32:57.117 INFO:teuthology.orchestra.run.smithi100.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 2.1 MB 00:01 2024-03-29T11:32:57.259 INFO:teuthology.orchestra.run.smithi100.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.7 MB/s | 247 kB 00:00 2024-03-29T11:32:57.442 INFO:teuthology.orchestra.run.smithi100.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-29T11:32:57.510 INFO:teuthology.orchestra.run.smithi100.stdout:(6/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 3.1 MB/s | 4.7 MB 00:01 2024-03-29T11:32:57.535 INFO:teuthology.orchestra.run.smithi100.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 144 kB 00:00 2024-03-29T11:32:57.560 INFO:teuthology.orchestra.run.smithi100.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 634 kB/s | 32 kB 00:00 2024-03-29T11:32:57.585 INFO:teuthology.orchestra.run.smithi100.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2024-03-29T11:32:57.611 INFO:teuthology.orchestra.run.smithi100.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-29T11:32:57.644 INFO:teuthology.orchestra.run.smithi100.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-29T11:32:57.803 INFO:teuthology.orchestra.run.smithi100.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 6.0 MB/s | 1.1 MB 00:00 2024-03-29T11:32:57.828 INFO:teuthology.orchestra.run.smithi100.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 561 kB/s | 103 kB 00:00 2024-03-29T11:32:57.854 INFO:teuthology.orchestra.run.smithi100.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:32:57.887 INFO:teuthology.orchestra.run.smithi100.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-29T11:32:57.996 INFO:teuthology.orchestra.run.smithi100.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 882 kB/s | 125 kB 00:00 2024-03-29T11:32:58.130 INFO:teuthology.orchestra.run.smithi100.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 92 kB/s | 22 kB 00:00 2024-03-29T11:32:58.430 INFO:teuthology.orchestra.run.smithi100.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 348 kB/s | 151 kB 00:00 2024-03-29T11:32:58.471 INFO:teuthology.orchestra.run.smithi100.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 127 kB/s | 43 kB 00:00 2024-03-29T11:32:58.597 INFO:teuthology.orchestra.run.smithi100.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-29T11:32:58.689 INFO:teuthology.orchestra.run.smithi100.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-03-29T11:32:58.789 INFO:teuthology.orchestra.run.smithi100.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.1 MB/s | 358 kB 00:00 2024-03-29T11:32:58.806 INFO:teuthology.orchestra.run.smithi100.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-03-29T11:32:58.881 INFO:teuthology.orchestra.run.smithi100.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-03-29T11:32:58.898 INFO:teuthology.orchestra.run.smithi100.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-29T11:32:58.990 INFO:teuthology.orchestra.run.smithi100.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-29T11:32:59.007 INFO:teuthology.orchestra.run.smithi100.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-03-29T11:32:59.082 INFO:teuthology.orchestra.run.smithi100.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 859 kB/s | 79 kB 00:00 2024-03-29T11:32:59.108 INFO:teuthology.orchestra.run.smithi100.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-29T11:32:59.200 INFO:teuthology.orchestra.run.smithi100.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-03-29T11:32:59.225 INFO:teuthology.orchestra.run.smithi100.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 768 kB/s | 90 kB 00:00 2024-03-29T11:32:59.317 INFO:teuthology.orchestra.run.smithi100.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 305 kB/s | 36 kB 00:00 2024-03-29T11:32:59.334 INFO:teuthology.orchestra.run.smithi100.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-29T11:32:59.459 INFO:teuthology.orchestra.run.smithi100.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 160 kB/s | 20 kB 00:00 2024-03-29T11:32:59.510 INFO:teuthology.orchestra.run.smithi100.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.2 MB/s | 427 kB 00:00 2024-03-29T11:33:02.668 INFO:teuthology.orchestra.run.smithi100.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 2.9 MB/s | 17 MB 00:05 2024-03-29T11:33:02.671 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:02.671 INFO:teuthology.orchestra.run.smithi100.stdout:Total 3.8 MB/s | 30 MB 00:07 2024-03-29T11:33:02.695 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:02.735 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:02.736 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:03.133 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:03.133 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:04.488 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:04.553 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:33:04.592 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:33:04.639 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:33:04.708 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:33:04.746 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:33:04.779 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:33:04.871 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:33:04.949 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:33:04.990 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:33:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:33:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:33:05.154 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:33:05.202 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:33:05.258 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:33:05.450 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:33:05.509 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:33:05.548 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:33:05.582 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:33:05.603 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:33:05.680 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:33:05.714 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:05.748 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:33:05.788 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:33:05.823 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:33:05.857 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:33:05.946 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:33:05.988 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:33:06.060 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:33:06.205 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:33:06.331 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:06.488 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:33:06.518 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:06.544 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:06.544 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:33:06.544 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:06.544 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:06.544 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:07.969 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:07.996 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:07.997 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:07.997 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:33:07.997 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:07.997 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:07.997 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:08.432 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:08.455 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:08.455 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:08.455 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:33:08.455 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:08.455 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:08.455 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:08.654 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:08.677 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:08.677 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:08.677 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:33:08.677 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:08.678 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:08.678 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:10.530 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:10.530 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:33:10.530 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:33:10.530 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:33:10.530 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:33:10.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:33:10.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:10.533 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:33:10.533 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:33:10.533 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:33:10.533 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:33:10.533 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:33:11.606 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:33:11.606 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:11.606 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:11.606 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:11.606 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:11.606 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:11.606 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:11.606 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:33:11.607 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:33:11.608 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:33:11.609 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:33:11.609 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:33:11.609 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:33:11.609 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:33:11.609 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:33:11.609 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:11.609 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:11.953 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-base 2024-03-29T11:33:12.542 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:12.640 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:12.692 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:12.693 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-29T11:33:12.693 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:12.768 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install cephadm 2024-03-29T11:33:13.328 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:13.478 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:13.479 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:33:13.479 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:13.479 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:33:13.479 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:33:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:13.480 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:33:13.480 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:13.480 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-29T11:33:13.480 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:13.480 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 747 k 2024-03-29T11:33:13.480 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 754 k 2024-03-29T11:33:13.480 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:33:13.964 INFO:teuthology.orchestra.run.smithi100.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.5 MB/s | 747 kB 00:00 2024-03-29T11:33:13.965 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:13.965 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.5 MB/s | 747 kB 00:00 2024-03-29T11:33:13.965 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:13.970 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:13.970 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:13.976 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:13.976 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:14.046 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:14.522 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:14.601 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:14.940 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:15.262 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:15.262 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:15.263 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:15.263 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:15.263 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:15.263 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:15.481 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:33:16.044 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:16.198 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:16.199 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:16.199 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-03-29T11:33:16.199 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:16.199 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:33:16.199 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:33:16.199 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:16.200 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:33:16.200 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:16.200 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-29T11:33:16.200 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:16.200 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 145 k 2024-03-29T11:33:16.200 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 436 k 2024-03-29T11:33:16.200 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:33:16.545 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 422 kB/s | 145 kB 00:00 2024-03-29T11:33:16.545 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:16.545 INFO:teuthology.orchestra.run.smithi100.stdout:Total 420 kB/s | 145 kB 00:00 2024-03-29T11:33:16.545 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:16.553 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:16.553 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:16.605 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:16.605 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:16.743 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:16.804 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:16.827 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:16.828 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:16.828 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:33:16.828 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:17.806 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:17.806 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:17.807 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:17.807 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.807 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:17.807 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:18.042 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr 2024-03-29T11:33:18.614 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:18.713 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:18.766 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:18.767 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-29T11:33:18.767 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:18.842 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:33:19.400 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:19.556 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:19.556 INFO:teuthology.orchestra.run.smithi100.stdout:====================================================================================== 2024-03-29T11:33:19.556 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:33:19.556 INFO:teuthology.orchestra.run.smithi100.stdout:====================================================================================== 2024-03-29T11:33:19.556 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:33:19.557 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:33:19.558 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:19.558 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:33:19.558 INFO:teuthology.orchestra.run.smithi100.stdout:====================================================================================== 2024-03-29T11:33:19.558 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-03-29T11:33:19.558 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:19.558 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 6.9 M 2024-03-29T11:33:19.558 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 26 M 2024-03-29T11:33:19.558 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:33:19.966 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 175 kB/s | 31 kB 00:00 2024-03-29T11:33:19.991 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 75 kB/s | 15 kB 00:00 2024-03-29T11:33:20.133 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T11:33:20.183 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-29T11:33:20.401 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.9 MB/s | 1.8 MB 00:00 2024-03-29T11:33:20.634 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T11:33:20.659 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-29T11:33:20.835 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 10 MB/s | 2.0 MB 00:00 2024-03-29T11:33:20.860 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 721 kB/s | 144 kB 00:00 2024-03-29T11:33:20.893 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 525 kB/s | 31 kB 00:00 2024-03-29T11:33:20.977 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-29T11:33:21.835 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 507 kB/s | 863 kB 00:01 2024-03-29T11:33:21.843 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:21.843 INFO:teuthology.orchestra.run.smithi100.stdout:Total 3.0 MB/s | 6.9 MB 00:02 2024-03-29T11:33:21.864 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:21.882 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:21.883 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:22.061 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:22.062 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:22.577 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:22.644 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:33:22.687 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:33:22.734 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:33:22.773 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:33:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:33:22.997 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:33:23.092 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:33:23.136 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:33:23.165 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:23.196 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:33:23.440 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:33:23.463 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:24.333 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:24.333 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:33:24.333 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:33:24.333 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:33:24.334 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:33:24.334 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:33:24.334 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:33:24.334 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:33:24.334 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:33:24.334 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:24.334 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:33:24.334 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:33:24.696 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:33:24.696 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:24.696 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:24.696 INFO:teuthology.orchestra.run.smithi100.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:33:24.696 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:24.696 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:24.696 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:24.697 INFO:teuthology.orchestra.run.smithi100.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:33:24.697 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:33:24.697 INFO:teuthology.orchestra.run.smithi100.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:33:24.697 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:33:24.697 INFO:teuthology.orchestra.run.smithi100.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:33:24.697 INFO:teuthology.orchestra.run.smithi100.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:33:24.697 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:33:24.697 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:33:24.697 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:24.697 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:24.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:33:25.522 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:25.676 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout:============================================================================================ 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout:============================================================================================ 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:33:25.677 INFO:teuthology.orchestra.run.smithi100.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:33:25.678 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:33:25.678 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:33:25.678 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:33:25.678 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:33:25.678 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:33:25.678 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:25.678 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:33:25.678 INFO:teuthology.orchestra.run.smithi100.stdout:============================================================================================ 2024-03-29T11:33:25.678 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-03-29T11:33:25.678 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:25.679 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 37 M 2024-03-29T11:33:25.679 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 209 M 2024-03-29T11:33:25.679 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:33:25.960 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 247 kB/s | 32 kB 00:00 2024-03-29T11:33:25.994 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-29T11:33:26.060 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T11:33:26.086 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 380 kB/s | 34 kB 00:00 2024-03-29T11:33:26.278 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-29T11:33:26.345 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T11:33:26.370 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T11:33:26.571 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 803 kB 00:00 2024-03-29T11:33:26.722 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-29T11:33:26.746 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T11:33:26.863 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-29T11:33:27.821 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.7 MB/s | 7.4 MB 00:01 2024-03-29T11:33:27.824 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:27.824 INFO:teuthology.orchestra.run.smithi100.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-29T11:33:27.868 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:27.893 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:27.893 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:28.255 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:28.255 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:28.969 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:29.044 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:29.075 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:29.168 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:29.212 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:29.388 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:29.436 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:29.506 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:30.180 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:30.290 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:31.535 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:32.563 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:32.586 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:35.134 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:35.135 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:35.602 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:35.602 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:35.602 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:35.602 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:35.602 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:35.602 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:35.602 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:35.603 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:35.603 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:35.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:36.547 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:36.698 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:36.699 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================== 2024-03-29T11:33:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:33:36.699 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================== 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:36.701 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:33:36.701 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================== 2024-03-29T11:33:36.701 INFO:teuthology.orchestra.run.smithi100.stdout:Install 13 Packages 2024-03-29T11:33:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:36.702 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 2.0 M 2024-03-29T11:33:36.702 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 26 M 2024-03-29T11:33:36.702 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:33:37.153 INFO:teuthology.orchestra.run.smithi100.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 232 kB/s | 24 kB 00:00 2024-03-29T11:33:37.194 INFO:teuthology.orchestra.run.smithi100.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-29T11:33:37.278 INFO:teuthology.orchestra.run.smithi100.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-29T11:33:37.337 INFO:teuthology.orchestra.run.smithi100.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 931 kB/s | 132 kB 00:00 2024-03-29T11:33:37.404 INFO:teuthology.orchestra.run.smithi100.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 211 kB 00:00 2024-03-29T11:33:37.421 INFO:teuthology.orchestra.run.smithi100.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 510 kB/s | 43 kB 00:00 2024-03-29T11:33:37.438 INFO:teuthology.orchestra.run.smithi100.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 134 kB/s | 52 kB 00:00 2024-03-29T11:33:37.688 INFO:teuthology.orchestra.run.smithi100.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-03-29T11:33:37.713 INFO:teuthology.orchestra.run.smithi100.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-03-29T11:33:37.855 INFO:teuthology.orchestra.run.smithi100.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 486 kB/s | 203 kB 00:00 2024-03-29T11:33:37.880 INFO:teuthology.orchestra.run.smithi100.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-29T11:33:37.981 INFO:teuthology.orchestra.run.smithi100.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 716 kB/s | 90 kB 00:00 2024-03-29T11:33:38.014 INFO:teuthology.orchestra.run.smithi100.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-29T11:33:38.018 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:38.018 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T11:33:38.019 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:38.031 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:38.031 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:38.121 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:38.122 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:38.636 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:38.810 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:38.865 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:38.921 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:38.958 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:38.999 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:39.102 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:39.211 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:39.259 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:39.979 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:40.023 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:40.057 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:40.105 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:40.127 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:40.939 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:40.939 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:40.939 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:40.939 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:40.939 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:40.939 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:40.939 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:40.939 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:40.940 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:40.940 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:40.940 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:40.940 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:40.940 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:41.298 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:41.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:41.300 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:41.300 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:41.300 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:41.300 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:41.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:42.138 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:42.290 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:42.291 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================== 2024-03-29T11:33:42.291 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:33:42.291 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================== 2024-03-29T11:33:42.291 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:33:42.291 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================== 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout:Install 6 Packages 2024-03-29T11:33:42.292 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:42.293 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 6.8 M 2024-03-29T11:33:42.293 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 33 M 2024-03-29T11:33:42.293 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:33:43.133 INFO:teuthology.orchestra.run.smithi100.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 534 kB/s | 138 kB 00:00 2024-03-29T11:33:43.208 INFO:teuthology.orchestra.run.smithi100.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-29T11:33:43.383 INFO:teuthology.orchestra.run.smithi100.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 487 kB/s | 248 kB 00:00 2024-03-29T11:33:43.558 INFO:teuthology.orchestra.run.smithi100.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-03-29T11:33:43.734 INFO:teuthology.orchestra.run.smithi100.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 146 kB/s | 51 kB 00:00 2024-03-29T11:33:43.800 INFO:teuthology.orchestra.run.smithi100.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 821 kB/s | 548 kB 00:00 2024-03-29T11:33:43.802 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:43.802 INFO:teuthology.orchestra.run.smithi100.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2024-03-29T11:33:43.817 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:43.828 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:43.828 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:43.918 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:43.919 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:44.215 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:44.392 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:45.029 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:45.091 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:45.921 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:45.922 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:45.922 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:45.922 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:45.922 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:45.922 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:46.223 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:46.223 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:46.223 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:46.223 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:46.223 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:46.223 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:46.224 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:46.224 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:46.224 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:46.224 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:46.224 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:46.475 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-fuse 2024-03-29T11:33:47.033 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:06 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:47.182 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:47.183 INFO:teuthology.orchestra.run.smithi100.stdout:Install 2 Packages 2024-03-29T11:33:47.184 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:47.184 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 964 k 2024-03-29T11:33:47.184 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 2.8 M 2024-03-29T11:33:47.184 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:33:47.310 INFO:teuthology.orchestra.run.smithi100.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 625 kB/s | 79 kB 00:00 2024-03-29T11:33:47.569 INFO:teuthology.orchestra.run.smithi100.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 885 kB 00:00 2024-03-29T11:33:47.571 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:47.571 INFO:teuthology.orchestra.run.smithi100.stdout:Total 2.4 MB/s | 964 kB 00:00 2024-03-29T11:33:47.571 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:47.591 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:47.591 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:47.664 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:47.665 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:47.850 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:47.936 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 157 kB/s | 21 MB 02:15 2024-03-29T11:33:47.986 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:48.052 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:48.801 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:48.801 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:49.105 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:49.105 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:49.105 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:49.105 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:49.105 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:49.105 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:49.364 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-volume 2024-03-29T11:33:49.943 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:50.099 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:50.100 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:50.100 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-29T11:33:50.100 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:50.100 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:33:50.100 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:50.100 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-29T11:33:50.100 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:50.100 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:50.100 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:33:50.101 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:50.101 INFO:teuthology.orchestra.run.smithi100.stdout:Install 2 Packages 2024-03-29T11:33:50.101 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:50.101 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 332 k 2024-03-29T11:33:50.101 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 1.5 M 2024-03-29T11:33:50.101 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:33:50.216 INFO:teuthology.orchestra.run.smithi100.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 601 kB/s | 69 kB 00:00 2024-03-29T11:33:50.499 INFO:teuthology.orchestra.run.smithi100.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 662 kB/s | 264 kB 00:00 2024-03-29T11:33:50.500 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:50.501 INFO:teuthology.orchestra.run.smithi100.stdout:Total 829 kB/s | 332 kB 00:00 2024-03-29T11:33:50.501 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:50.510 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:50.510 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:50.547 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:50.547 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:50.692 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:50.864 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:50.902 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:50.918 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:50.919 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:50.919 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:50.919 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:51.732 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:52.062 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:33:52.063 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:52.063 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:52.063 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:52.063 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:33:52.063 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:52.063 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:52.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados-devel 2024-03-29T11:33:52.867 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-29T11:33:52.990 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:53.140 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:53.141 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:53.141 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-03-29T11:33:53.141 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:53.141 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:33:53.141 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:33:53.141 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:53.141 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:33:53.141 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:53.141 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-29T11:33:53.142 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:53.142 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 127 k 2024-03-29T11:33:53.142 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 456 k 2024-03-29T11:33:53.142 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:33:53.366 INFO:teuthology.orchestra.run.smithi100.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 566 kB/s | 127 kB 00:00 2024-03-29T11:33:53.366 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:53.367 INFO:teuthology.orchestra.run.smithi100.stdout:Total 563 kB/s | 127 kB 00:00 2024-03-29T11:33:53.367 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:53.372 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:53.372 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:53.427 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:53.427 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:53.579 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:53.682 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:54.129 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:54.460 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:54.460 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:54.460 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:54.460 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:54.460 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:54.460 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:54.786 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs2 2024-03-29T11:33:55.369 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:55.470 INFO:teuthology.orchestra.run.smithi100.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:55.481 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:33:55.507 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:33:55.507 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:33:55.507 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:33:55.507 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:33:55.507 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:33:55.508 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:33:55.509 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout:Install 30 Packages 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2024-03-29T11:33:55.510 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:33:55.511 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 58 M 2024-03-29T11:33:55.511 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:33:55.526 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:55.527 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-29T11:33:55.527 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:55.610 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs-devel 2024-03-29T11:33:56.174 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:56.326 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:56.327 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:56.327 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-03-29T11:33:56.327 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:56.327 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:33:56.327 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:33:56.327 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:56.327 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:33:56.327 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:33:56.328 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-29T11:33:56.328 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:56.328 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 31 k 2024-03-29T11:33:56.328 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 140 k 2024-03-29T11:33:56.328 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:33:56.612 INFO:teuthology.orchestra.run.smithi100.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 110 kB/s | 31 kB 00:00 2024-03-29T11:33:56.613 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:56.613 INFO:teuthology.orchestra.run.smithi100.stdout:Total 109 kB/s | 31 kB 00:00 2024-03-29T11:33:56.613 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:33:56.616 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:33:56.616 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:33:56.648 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:33:56.648 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:33:56.786 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:33:56.868 INFO:teuthology.orchestra.run.smithi191.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 9.9 MB/s | 9.7 MB 00:00 2024-03-29T11:33:56.903 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:56.927 INFO:teuthology.orchestra.run.smithi191.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 431 kB/s | 25 kB 00:00 2024-03-29T11:33:57.011 INFO:teuthology.orchestra.run.smithi191.stdout:(3/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.7 MB/s | 5.3 MB 00:01 2024-03-29T11:33:57.053 INFO:teuthology.orchestra.run.smithi191.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.8 MB/s | 742 kB 00:00 2024-03-29T11:33:57.087 INFO:teuthology.orchestra.run.smithi191.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 6.5 MB/s | 498 kB 00:00 2024-03-29T11:33:57.145 INFO:teuthology.orchestra.run.smithi191.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 772 kB/s | 45 kB 00:00 2024-03-29T11:33:57.229 INFO:teuthology.orchestra.run.smithi191.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-29T11:33:57.289 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:57.321 INFO:teuthology.orchestra.run.smithi191.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.7 MB/s | 162 kB 00:00 2024-03-29T11:33:57.430 INFO:teuthology.orchestra.run.smithi191.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-03-29T11:33:57.455 INFO:teuthology.orchestra.run.smithi191.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.3 MB/s | 320 kB 00:00 2024-03-29T11:33:57.489 INFO:teuthology.orchestra.run.smithi191.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T11:33:57.522 INFO:teuthology.orchestra.run.smithi191.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 99 kB 00:00 2024-03-29T11:33:57.563 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:57.563 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:57.563 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:33:57.563 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:57.563 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:33:57.564 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:57.640 INFO:teuthology.orchestra.run.smithi191.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 703 kB/s | 106 kB 00:00 2024-03-29T11:33:57.665 INFO:teuthology.orchestra.run.smithi191.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-29T11:33:57.690 INFO:teuthology.orchestra.run.smithi191.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-29T11:33:57.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados2 2024-03-29T11:33:57.741 INFO:teuthology.orchestra.run.smithi191.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T11:33:57.766 INFO:teuthology.orchestra.run.smithi191.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T11:33:57.792 INFO:teuthology.orchestra.run.smithi191.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-03-29T11:33:57.817 INFO:teuthology.orchestra.run.smithi191.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:33:57.834 INFO:teuthology.orchestra.run.smithi191.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 831 kB/s | 35 kB 00:00 2024-03-29T11:33:57.851 INFO:teuthology.orchestra.run.smithi191.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.3 MB/s | 46 kB 00:00 2024-03-29T11:33:58.162 INFO:teuthology.orchestra.run.smithi191.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 8.7 MB/s | 20 MB 00:02 2024-03-29T11:33:58.299 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:58.396 INFO:teuthology.orchestra.run.smithi100.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:58.402 INFO:teuthology.orchestra.run.smithi191.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 108 kB/s | 26 kB 00:00 2024-03-29T11:33:58.436 INFO:teuthology.orchestra.run.smithi191.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 514 kB/s | 309 kB 00:00 2024-03-29T11:33:58.447 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:58.448 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-29T11:33:58.448 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:58.504 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librbd1 2024-03-29T11:33:58.536 INFO:teuthology.orchestra.run.smithi191.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-03-29T11:33:58.553 INFO:teuthology.orchestra.run.smithi191.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 321 kB/s | 49 kB 00:00 2024-03-29T11:33:58.704 INFO:teuthology.orchestra.run.smithi191.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-03-29T11:33:58.729 INFO:teuthology.orchestra.run.smithi191.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:33:58.754 INFO:teuthology.orchestra.run.smithi191.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-03-29T11:33:58.805 INFO:teuthology.orchestra.run.smithi191.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-29T11:33:59.055 INFO:teuthology.orchestra.run.smithi191.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 10 MB/s | 3.1 MB 00:00 2024-03-29T11:33:59.069 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:59.167 INFO:teuthology.orchestra.run.smithi100.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:59.221 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:59.223 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-29T11:33:59.223 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:33:59.255 INFO:teuthology.orchestra.run.smithi191.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 6.4 MB/s | 3.4 MB 00:00 2024-03-29T11:33:59.262 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:59.262 INFO:teuthology.orchestra.run.smithi191.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-29T11:33:59.262 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:33:59.279 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rados 2024-03-29T11:33:59.296 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:33:59.296 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:33:59.772 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:33:59.772 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:33:59.846 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:33:59.943 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:59.995 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:33:59.996 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-29T11:33:59.996 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:34:00.045 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rgw 2024-03-29T11:34:00.604 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:34:00.702 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:00.754 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:34:00.755 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-29T11:34:00.755 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:34:00.812 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-cephfs 2024-03-29T11:34:01.230 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:34:01.305 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:34:01.361 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:34:01.382 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:34:01.386 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:34:01.409 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:34:01.426 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:34:01.479 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:01.531 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:34:01.532 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-29T11:34:01.532 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:34:01.588 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rbd 2024-03-29T11:34:01.619 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:34:01.674 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:34:01.993 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:34:02.056 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:34:02.148 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:34:02.152 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:34:02.235 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:34:02.250 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:02.275 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:34:02.301 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:34:02.302 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-29T11:34:02.303 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:34:02.321 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:34:02.350 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:34:02.353 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-fuse 2024-03-29T11:34:02.394 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:34:02.470 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:34:02.504 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:34:02.550 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:34:02.586 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:34:02.693 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:34:02.737 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:34:02.778 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:34:02.811 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:34:02.840 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:34:02.873 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:34:02.912 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:34:03.061 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:34:03.061 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:34:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 85 k 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 231 k 2024-03-29T11:34:03.062 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:34:03.193 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:34:03.229 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:34:03.248 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 459 kB/s | 85 kB 00:00 2024-03-29T11:34:03.249 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:03.249 INFO:teuthology.orchestra.run.smithi100.stdout:Total 455 kB/s | 85 kB 00:00 2024-03-29T11:34:03.249 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:34:03.256 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:34:03.256 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:34:03.301 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:34:03.310 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:34:03.311 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:34:03.325 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:34:03.368 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:34:03.451 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:34:03.555 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:03.827 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:34:03.857 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:34:03.937 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:04.094 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:34:04.164 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:34:04.193 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:34:04.246 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:04.246 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:04.246 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:34:04.246 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:04.247 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:04.247 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:34:04.492 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-mirror 2024-03-29T11:34:05.068 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:34:05.224 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:34:05.224 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:34:05.225 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-03-29T11:34:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:34:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:34:05.225 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:34:05.225 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:34:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:34:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-29T11:34:05.225 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:05.226 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 3.1 M 2024-03-29T11:34:05.226 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 13 M 2024-03-29T11:34:05.226 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:34:05.752 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.9 MB/s | 3.1 MB 00:00 2024-03-29T11:34:05.753 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:05.753 INFO:teuthology.orchestra.run.smithi100.stdout:Total 5.9 MB/s | 3.1 MB 00:00 2024-03-29T11:34:05.753 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:34:05.762 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:34:05.762 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:34:05.835 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:34:05.835 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:34:06.264 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:34:06.319 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:06.325 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:34:06.343 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:06.375 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:06.375 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:34:06.375 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:06.375 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:06.376 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:06.376 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:34:06.932 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:34:06.980 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:34:07.075 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:34:07.075 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:34:07.076 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:34:07.076 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:07.115 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:34:07.529 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:07.529 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:07.529 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:34:07.529 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:07.529 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:07.529 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:34:07.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-nbd 2024-03-29T11:34:08.361 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:31:41 AM UTC. 2024-03-29T11:34:08.508 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-29T11:34:08.509 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:34:08.509 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:08.509 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:34:08.509 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-29T11:34:08.509 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:34:08.509 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:08.509 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-29T11:34:08.509 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-29T11:34:08.509 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-29T11:34:08.510 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:08.510 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 171 k 2024-03-29T11:34:08.510 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 494 k 2024-03-29T11:34:08.510 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-29T11:34:08.739 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 750 kB/s | 171 kB 00:00 2024-03-29T11:34:08.739 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:08.740 INFO:teuthology.orchestra.run.smithi100.stdout:Total 744 kB/s | 171 kB 00:00 2024-03-29T11:34:08.740 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-29T11:34:08.747 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-29T11:34:08.747 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-29T11:34:08.803 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-29T11:34:08.803 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-29T11:34:08.951 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-29T11:34:09.062 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:09.481 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:09.769 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-29T11:34:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:34:09.769 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-29T11:34:10.000 DEBUG:teuthology.parallel:result is None 2024-03-29T11:34:16.795 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:34:16.796 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-03-29T11:34:16.796 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-03-29T11:34:16.796 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-03-29T11:34:16.796 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-03-29T11:34:16.796 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-03-29T11:34:16.796 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-03-29T11:34:16.796 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-03-29T11:34:16.796 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:17.730 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:34:17.754 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:34:17.754 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:17.754 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:34:17.754 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:34:17.754 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:34:17.754 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:17.756 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:34:17.840 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:34:17.841 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:34:19.011 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:34:19.011 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:34:19.011 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:34:19.011 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:34:19.011 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:34:19.011 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:34:19.011 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:34:19.011 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:34:19.011 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:34:19.011 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:34:19.012 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:34:19.013 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:34:19.013 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:34:19.013 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:34:19.013 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:34:19.013 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:34:19.013 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:34:19.013 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:34:19.013 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:34:19.013 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.990 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:34:19.991 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:19.992 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:34:20.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-test 2024-03-29T11:34:20.784 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:34:20.935 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:34:20.935 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:34:20.935 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:34:20.935 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2024-03-29T11:34:20.936 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:20.937 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 48 M 2024-03-29T11:34:20.937 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 204 M 2024-03-29T11:34:20.937 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:34:21.252 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T11:34:21.311 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-29T11:34:21.336 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T11:34:21.486 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 395 kB/s | 163 kB 00:00 2024-03-29T11:34:21.536 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 934 kB/s | 187 kB 00:00 2024-03-29T11:34:24.154 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 15 MB/s | 47 MB 00:03 2024-03-29T11:34:24.156 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:24.156 INFO:teuthology.orchestra.run.smithi191.stdout:Total 15 MB/s | 48 MB 00:03 2024-03-29T11:34:24.171 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:34:24.192 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:34:24.192 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:34:24.571 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:34:24.571 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:34:25.142 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:34:25.220 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:34:25.261 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:34:25.300 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:34:25.336 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:34:25.373 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:34:29.855 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:34:29.997 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:34:32.342 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:34:32.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:34:32.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:34:32.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:34:32.342 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:34:32.343 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:34:32.699 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:34:32.699 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:32.699 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:34:32.700 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:32.700 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:34:32.700 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:32.700 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:34:32.700 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:34:32.700 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:34:32.700 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:32.700 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:34:32.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph 2024-03-29T11:34:33.591 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:34:33.762 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:34:33.764 INFO:teuthology.orchestra.run.smithi191.stdout:========================================================================================= 2024-03-29T11:34:33.764 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:34:33.764 INFO:teuthology.orchestra.run.smithi191.stdout:========================================================================================= 2024-03-29T11:34:33.764 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:34:33.765 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:34:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:34:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:34:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:34:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:34:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:34:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:34:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:34:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:34:33.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:34:33.770 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:34:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:34:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:34:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:34:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:34:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:33.771 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:34:33.771 INFO:teuthology.orchestra.run.smithi191.stdout:========================================================================================= 2024-03-29T11:34:33.771 INFO:teuthology.orchestra.run.smithi191.stdout:Install 36 Packages 2024-03-29T11:34:33.771 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:33.772 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 30 M 2024-03-29T11:34:33.772 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 107 M 2024-03-29T11:34:33.772 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:34:34.538 INFO:teuthology.orchestra.run.smithi191.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-03-29T11:34:34.772 INFO:teuthology.orchestra.run.smithi191.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.6 MB/s | 1.5 MB 00:00 2024-03-29T11:34:35.080 INFO:teuthology.orchestra.run.smithi191.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 8.6 MB/s | 4.7 MB 00:00 2024-03-29T11:34:35.131 INFO:teuthology.orchestra.run.smithi191.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 2.1 MB 00:00 2024-03-29T11:34:35.156 INFO:teuthology.orchestra.run.smithi191.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.2 MB/s | 247 kB 00:00 2024-03-29T11:34:35.281 INFO:teuthology.orchestra.run.smithi191.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:34:35.306 INFO:teuthology.orchestra.run.smithi191.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-29T11:34:35.332 INFO:teuthology.orchestra.run.smithi191.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 632 kB/s | 32 kB 00:00 2024-03-29T11:34:35.357 INFO:teuthology.orchestra.run.smithi191.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-03-29T11:34:35.382 INFO:teuthology.orchestra.run.smithi191.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-29T11:34:35.424 INFO:teuthology.orchestra.run.smithi191.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-29T11:34:35.533 INFO:teuthology.orchestra.run.smithi191.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 7.6 MB/s | 1.1 MB 00:00 2024-03-29T11:34:35.558 INFO:teuthology.orchestra.run.smithi191.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 774 kB/s | 103 kB 00:00 2024-03-29T11:34:35.583 INFO:teuthology.orchestra.run.smithi191.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:34:35.609 INFO:teuthology.orchestra.run.smithi191.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:34:35.834 INFO:teuthology.orchestra.run.smithi191.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 16 MB/s | 17 MB 00:01 2024-03-29T11:34:35.884 INFO:teuthology.orchestra.run.smithi191.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 416 kB/s | 125 kB 00:00 2024-03-29T11:34:35.918 INFO:teuthology.orchestra.run.smithi191.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 72 kB/s | 22 kB 00:00 2024-03-29T11:34:36.285 INFO:teuthology.orchestra.run.smithi191.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 108 kB/s | 43 kB 00:00 2024-03-29T11:34:36.393 INFO:teuthology.orchestra.run.smithi191.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 270 kB/s | 151 kB 00:00 2024-03-29T11:34:36.452 INFO:teuthology.orchestra.run.smithi191.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 322 kB/s | 172 kB 00:00 2024-03-29T11:34:36.494 INFO:teuthology.orchestra.run.smithi191.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-29T11:34:36.553 INFO:teuthology.orchestra.run.smithi191.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-03-29T11:34:36.594 INFO:teuthology.orchestra.run.smithi191.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-29T11:34:36.653 INFO:teuthology.orchestra.run.smithi191.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 974 kB/s | 358 kB 00:00 2024-03-29T11:34:36.678 INFO:teuthology.orchestra.run.smithi191.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-03-29T11:34:36.703 INFO:teuthology.orchestra.run.smithi191.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-29T11:34:36.754 INFO:teuthology.orchestra.run.smithi191.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-03-29T11:34:36.804 INFO:teuthology.orchestra.run.smithi191.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 629 kB/s | 79 kB 00:00 2024-03-29T11:34:36.854 INFO:teuthology.orchestra.run.smithi191.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-29T11:34:36.904 INFO:teuthology.orchestra.run.smithi191.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-29T11:34:36.929 INFO:teuthology.orchestra.run.smithi191.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 717 kB/s | 90 kB 00:00 2024-03-29T11:34:36.955 INFO:teuthology.orchestra.run.smithi191.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-03-29T11:34:37.030 INFO:teuthology.orchestra.run.smithi191.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-29T11:34:37.056 INFO:teuthology.orchestra.run.smithi191.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-29T11:34:37.189 INFO:teuthology.orchestra.run.smithi191.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-03-29T11:34:37.196 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:37.196 INFO:teuthology.orchestra.run.smithi191.stdout:Total 8.7 MB/s | 30 MB 00:03 2024-03-29T11:34:37.221 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:34:37.263 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:34:37.263 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:34:37.672 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:34:37.672 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:34:39.044 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:34:39.092 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:34:39.135 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:34:39.161 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:34:39.232 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:34:39.293 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:34:39.334 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:34:39.426 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:34:39.512 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:34:39.562 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:34:39.633 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:34:39.687 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:34:39.725 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:34:39.774 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:34:39.838 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:34:40.022 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:34:40.071 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:34:40.102 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:34:40.137 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:34:40.166 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:34:40.242 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:34:40.277 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:34:40.310 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:34:40.350 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:34:40.385 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:34:40.420 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:34:40.509 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:34:40.543 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:34:40.623 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:34:40.760 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:34:40.877 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:34:41.042 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:34:41.064 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:34:41.090 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:34:41.090 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:41.090 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:34:41.090 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:34:41.090 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:34:41.090 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:42.532 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:34:42.558 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:34:42.559 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:42.559 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:34:42.559 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:34:42.559 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:34:42.559 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:42.999 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:34:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:34:43.023 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:43.023 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:34:43.023 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:34:43.024 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:34:43.024 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:43.233 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:34:43.257 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:34:43.257 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:43.257 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:34:43.257 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:34:43.257 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:34:43.257 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:43.322 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:34:44.907 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:34:44.907 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:34:44.907 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:34:44.907 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:34:44.907 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:34:44.907 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:34:44.907 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:34:44.908 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:34:44.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:34:44.910 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:34:44.910 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:34:45.844 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:34:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:45.845 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:34:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:34:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:34:45.845 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:34:45.846 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:34:45.847 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:34:45.848 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:45.848 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:34:46.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base 2024-03-29T11:34:46.766 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:34:46.863 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:46.916 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:34:46.917 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-29T11:34:46.917 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:34:46.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install cephadm 2024-03-29T11:34:47.538 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:34:47.689 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:34:47.690 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:34:47.690 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:34:47.690 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:34:47.690 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:34:47.690 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:34:47.690 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:47.690 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:34:47.690 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:34:47.690 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-29T11:34:47.691 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:47.691 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 747 k 2024-03-29T11:34:47.691 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 754 k 2024-03-29T11:34:47.691 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:34:48.141 INFO:teuthology.orchestra.run.smithi191.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.6 MB/s | 747 kB 00:00 2024-03-29T11:34:48.141 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:48.142 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.6 MB/s | 747 kB 00:00 2024-03-29T11:34:48.142 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:34:48.147 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:34:48.147 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:34:48.152 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:34:48.153 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:34:48.224 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:34:48.674 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:34:48.761 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:34:49.143 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:34:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:34:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:49.430 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:34:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:34:49.430 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:49.430 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:34:49.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:34:50.174 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:34:50.331 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:34:50.332 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:34:50.332 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-29T11:34:50.332 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:34:50.332 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:34:50.332 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:34:50.332 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:50.333 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:34:50.333 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:34:50.333 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-29T11:34:50.333 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:50.333 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 145 k 2024-03-29T11:34:50.333 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 436 k 2024-03-29T11:34:50.333 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:34:50.555 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 655 kB/s | 145 kB 00:00 2024-03-29T11:34:50.556 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:50.556 INFO:teuthology.orchestra.run.smithi191.stdout:Total 650 kB/s | 145 kB 00:00 2024-03-29T11:34:50.556 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:34:50.563 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:34:50.563 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:34:50.616 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:34:50.616 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:34:50.756 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:34:50.823 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:34:50.847 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:34:50.847 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:50.848 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:34:50.848 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:34:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:51.809 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:34:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:51.809 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:51.809 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:34:52.042 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr 2024-03-29T11:34:52.626 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:34:52.725 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:52.778 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:34:52.779 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-29T11:34:52.779 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:34:52.848 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:34:53.405 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:34:53.554 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================== 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================== 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:34:53.556 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:34:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:34:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:34:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:34:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:34:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:53.557 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:34:53.557 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================== 2024-03-29T11:34:53.557 INFO:teuthology.orchestra.run.smithi191.stdout:Install 12 Packages 2024-03-29T11:34:53.557 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:53.558 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 6.9 M 2024-03-29T11:34:53.558 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 26 M 2024-03-29T11:34:53.558 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:34:54.034 INFO:teuthology.orchestra.run.smithi191.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 160 kB/s | 31 kB 00:00 2024-03-29T11:34:54.143 INFO:teuthology.orchestra.run.smithi191.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 50 kB/s | 15 kB 00:00 2024-03-29T11:34:54.210 INFO:teuthology.orchestra.run.smithi191.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-29T11:34:54.235 INFO:teuthology.orchestra.run.smithi191.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-29T11:34:54.285 INFO:teuthology.orchestra.run.smithi191.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.0 MB/s | 1.8 MB 00:00 2024-03-29T11:34:54.411 INFO:teuthology.orchestra.run.smithi191.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 156 kB/s | 19 kB 00:00 2024-03-29T11:34:54.444 INFO:teuthology.orchestra.run.smithi191.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-03-29T11:34:54.544 INFO:teuthology.orchestra.run.smithi191.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-03-29T11:34:54.595 INFO:teuthology.orchestra.run.smithi191.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 613 kB/s | 31 kB 00:00 2024-03-29T11:34:54.628 INFO:teuthology.orchestra.run.smithi191.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 785 kB/s | 144 kB 00:00 2024-03-29T11:34:54.653 INFO:teuthology.orchestra.run.smithi191.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-29T11:34:54.962 INFO:teuthology.orchestra.run.smithi191.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.7 MB/s | 2.0 MB 00:00 2024-03-29T11:34:54.965 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:54.965 INFO:teuthology.orchestra.run.smithi191.stdout:Total 4.9 MB/s | 6.9 MB 00:01 2024-03-29T11:34:54.984 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:34:55.001 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:34:55.002 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:34:55.176 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:34:55.177 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:34:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:34:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:34:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:34:55.844 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:34:55.875 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:34:55.979 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:34:56.107 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:34:56.218 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:34:56.262 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:34:56.291 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:34:56.323 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:34:56.564 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:34:56.589 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:34:57.394 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:34:57.394 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:34:57.394 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:34:57.395 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:34:57.395 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:34:57.395 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:34:57.395 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:34:57.395 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:34:57.395 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:34:57.395 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:34:57.395 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:34:57.395 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:34:57.748 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:34:57.749 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:34:57.750 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:34:57.750 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:57.750 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:34:58.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:34:58.586 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:34:58.737 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:34:58.739 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:34:58.740 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:34:58.740 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:34:58.740 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:34:58.740 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:34:58.740 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:34:58.740 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:58.740 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:34:58.740 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-29T11:34:58.740 INFO:teuthology.orchestra.run.smithi191.stdout:Install 12 Packages 2024-03-29T11:34:58.740 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:34:58.741 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 37 M 2024-03-29T11:34:58.741 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 209 M 2024-03-29T11:34:58.741 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:34:59.260 INFO:teuthology.orchestra.run.smithi191.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 227 kB/s | 32 kB 00:00 2024-03-29T11:34:59.286 INFO:teuthology.orchestra.run.smithi191.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 721 kB/s | 17 kB 00:00 2024-03-29T11:34:59.319 INFO:teuthology.orchestra.run.smithi191.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:34:59.369 INFO:teuthology.orchestra.run.smithi191.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-29T11:34:59.570 INFO:teuthology.orchestra.run.smithi191.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-29T11:34:59.645 INFO:teuthology.orchestra.run.smithi191.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-29T11:34:59.670 INFO:teuthology.orchestra.run.smithi191.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T11:35:00.037 INFO:teuthology.orchestra.run.smithi191.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-29T11:35:00.146 INFO:teuthology.orchestra.run.smithi191.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 7.2 MB/s | 7.4 MB 00:01 2024-03-29T11:35:00.221 INFO:teuthology.orchestra.run.smithi191.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-29T11:35:00.288 INFO:teuthology.orchestra.run.smithi191.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 781 kB/s | 195 kB 00:00 2024-03-29T11:35:00.430 INFO:teuthology.orchestra.run.smithi191.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 866 kB/s | 245 kB 00:00 2024-03-29T11:35:00.432 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:35:00.432 INFO:teuthology.orchestra.run.smithi191.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-29T11:35:00.472 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:35:00.496 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:35:00.496 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:35:00.866 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:35:00.866 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:35:01.562 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:35:01.636 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:35:01.667 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:35:01.760 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:35:01.795 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:35:01.972 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:35:02.003 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:35:02.757 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:35:02.867 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:35:04.102 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:35:05.130 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:35:05.162 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:35:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:35:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:35:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:35:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:35:07.719 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:35:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:35:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:35:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:35:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:35:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:35:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:35:07.720 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:35:08.207 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:35:08.208 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:08.208 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:35:08.208 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:35:08.208 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:35:08.208 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:35:08.209 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:35:08.209 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:35:08.209 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:35:08.209 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:35:08.210 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:35:08.210 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:35:08.210 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:35:08.210 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:35:08.210 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:35:08.211 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:08.211 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:08.640 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:35:09.210 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:09.360 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:09.361 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-29T11:35:09.361 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:35:09.361 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-29T11:35:09.361 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:35:09.361 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:35:09.362 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:35:09.363 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:09.363 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:35:09.363 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-29T11:35:09.363 INFO:teuthology.orchestra.run.smithi191.stdout:Install 13 Packages 2024-03-29T11:35:09.363 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:09.363 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 2.0 M 2024-03-29T11:35:09.364 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 26 M 2024-03-29T11:35:09.364 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:35:09.596 INFO:teuthology.orchestra.run.smithi191.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 160 kB/s | 24 kB 00:00 2024-03-29T11:35:09.622 INFO:teuthology.orchestra.run.smithi191.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 102 kB/s | 18 kB 00:00 2024-03-29T11:35:09.672 INFO:teuthology.orchestra.run.smithi191.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:35:09.697 INFO:teuthology.orchestra.run.smithi191.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:35:09.722 INFO:teuthology.orchestra.run.smithi191.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:35:09.747 INFO:teuthology.orchestra.run.smithi191.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T11:35:09.773 INFO:teuthology.orchestra.run.smithi191.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 159 kB/s | 52 kB 00:00 2024-03-29T11:35:10.132 INFO:teuthology.orchestra.run.smithi191.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 39 kB/s | 15 kB 00:00 2024-03-29T11:35:10.173 INFO:teuthology.orchestra.run.smithi191.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 71 kB/s | 32 kB 00:00 2024-03-29T11:35:10.382 INFO:teuthology.orchestra.run.smithi191.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 283 kB/s | 59 kB 00:00 2024-03-29T11:35:10.407 INFO:teuthology.orchestra.run.smithi191.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 320 kB/s | 203 kB 00:00 2024-03-29T11:35:10.549 INFO:teuthology.orchestra.run.smithi191.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 537 kB/s | 90 kB 00:00 2024-03-29T11:35:10.699 INFO:teuthology.orchestra.run.smithi191.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-29T11:35:10.703 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:35:10.704 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-29T11:35:10.704 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:35:10.718 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:35:10.718 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:35:10.813 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:35:10.814 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:35:11.334 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:35:11.504 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:35:11.559 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:35:11.614 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:35:11.660 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:35:11.701 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:35:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:35:11.912 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:35:11.969 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:35:12.682 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:35:12.733 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:35:12.767 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:35:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:35:12.836 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:35:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:35:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:35:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:35:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:35:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:35:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:35:13.685 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:35:13.686 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:35:13.686 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:35:13.686 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:35:13.686 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:35:13.686 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:35:13.686 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:35:14.054 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:35:14.054 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:14.054 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:35:14.054 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:35:14.054 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:35:14.054 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:35:14.054 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:35:14.054 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:35:14.054 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:35:14.054 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:35:14.055 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:35:14.055 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:35:14.055 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:35:14.055 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:35:14.055 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:35:14.055 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:35:14.055 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:14.055 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:14.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:35:14.875 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:15.027 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:15.028 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-29T11:35:15.028 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:35:15.028 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-29T11:35:15.028 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2024-03-29T11:35:15.029 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:15.030 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 6.8 M 2024-03-29T11:35:15.030 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 33 M 2024-03-29T11:35:15.030 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:35:15.701 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T11:35:15.826 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 350 kB/s | 138 kB 00:00 2024-03-29T11:35:16.101 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 210 kB/s | 58 kB 00:00 2024-03-29T11:35:16.135 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-29T11:35:16.168 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 768 kB/s | 51 kB 00:00 2024-03-29T11:35:16.394 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 258 kB/s | 248 kB 00:00 2024-03-29T11:35:16.397 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:35:16.397 INFO:teuthology.orchestra.run.smithi191.stdout:Total 5.0 MB/s | 6.8 MB 00:01 2024-03-29T11:35:16.411 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:35:16.422 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:35:16.422 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:35:16.511 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:35:16.511 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:35:16.812 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:35:16.973 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:35:17.028 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:35:17.550 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:35:17.638 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:35:17.700 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:35:17.724 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:35:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:35:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:35:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:35:18.602 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:35:18.603 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:35:18.603 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:35:18.901 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:35:18.901 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:18.901 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:35:18.901 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:35:18.901 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:35:18.901 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:35:18.901 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:35:18.901 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:35:18.901 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:35:18.901 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:18.902 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:19.149 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-fuse 2024-03-29T11:35:19.713 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:27 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:19.864 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:19.864 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 964 k 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.8 M 2024-03-29T11:35:19.865 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:35:19.987 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 648 kB/s | 79 kB 00:00 2024-03-29T11:35:20.170 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 885 kB 00:00 2024-03-29T11:35:20.171 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:35:20.172 INFO:teuthology.orchestra.run.smithi191.stdout:Total 3.1 MB/s | 964 kB 00:00 2024-03-29T11:35:20.172 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:35:20.192 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:35:20.192 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:35:20.265 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:35:20.265 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:35:20.454 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:35:20.604 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:35:20.670 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:35:21.322 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:35:21.322 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:35:21.617 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:35:21.617 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:21.618 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:35:21.618 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:35:21.618 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:21.618 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:21.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-volume 2024-03-29T11:35:22.495 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:22.650 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:22.651 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:22.651 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-29T11:35:22.651 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:22.651 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 332 k 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 1.5 M 2024-03-29T11:35:22.652 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:35:22.775 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 565 kB/s | 69 kB 00:00 2024-03-29T11:35:23.017 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 725 kB/s | 264 kB 00:00 2024-03-29T11:35:23.018 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:35:23.018 INFO:teuthology.orchestra.run.smithi191.stdout:Total 907 kB/s | 332 kB 00:00 2024-03-29T11:35:23.018 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:35:23.027 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:35:23.027 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:35:23.067 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:35:23.067 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:35:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:35:23.380 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:35:23.411 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:35:23.427 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:35:23.428 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:35:23.428 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:35:23.428 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:24.240 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:35:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:35:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:24.558 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:35:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:35:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:35:24.558 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:24.558 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:24.874 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados-devel 2024-03-29T11:35:25.452 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:25.602 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:25.603 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:25.603 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-29T11:35:25.603 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:25.603 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:35:25.604 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:35:25.604 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:25.604 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:35:25.604 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:25.604 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-29T11:35:25.604 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:25.604 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 127 k 2024-03-29T11:35:25.604 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 456 k 2024-03-29T11:35:25.604 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:35:25.831 INFO:teuthology.orchestra.run.smithi191.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 559 kB/s | 127 kB 00:00 2024-03-29T11:35:25.831 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:35:25.832 INFO:teuthology.orchestra.run.smithi191.stdout:Total 555 kB/s | 127 kB 00:00 2024-03-29T11:35:25.832 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:35:25.837 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:35:25.837 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:35:25.893 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:35:25.893 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:35:26.049 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:35:26.138 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:26.603 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:26.876 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:26.877 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:26.877 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:35:26.877 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:35:26.877 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:26.877 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:27.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs2 2024-03-29T11:35:27.684 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:27.786 INFO:teuthology.orchestra.run.smithi191.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:35:27.843 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:27.843 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-29T11:35:27.844 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:27.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs-devel 2024-03-29T11:35:28.473 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:28.624 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:28.624 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:28.624 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-29T11:35:28.625 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:28.625 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:35:28.625 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:35:28.625 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:28.625 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:35:28.625 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:28.625 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-29T11:35:28.625 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:28.626 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 31 k 2024-03-29T11:35:28.626 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 140 k 2024-03-29T11:35:28.626 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:35:28.868 INFO:teuthology.orchestra.run.smithi191.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 129 kB/s | 31 kB 00:00 2024-03-29T11:35:28.868 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:35:28.869 INFO:teuthology.orchestra.run.smithi191.stdout:Total 128 kB/s | 31 kB 00:00 2024-03-29T11:35:28.869 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:35:28.872 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:35:28.872 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:35:28.894 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:35:28.895 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:35:29.005 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:35:29.100 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:29.492 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:29.774 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:29.774 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:29.774 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:35:29.774 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:35:29.774 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:29.775 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:29.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados2 2024-03-29T11:35:30.498 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:30.595 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:35:30.646 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:30.647 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-29T11:35:30.647 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:30.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librbd1 2024-03-29T11:35:31.253 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:31.354 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:35:31.408 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:31.409 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-29T11:35:31.409 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:31.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rados 2024-03-29T11:35:32.027 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:32.125 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:35:32.177 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:32.178 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-29T11:35:32.178 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:32.229 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rgw 2024-03-29T11:35:32.793 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:32.892 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:35:32.944 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:32.945 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-29T11:35:32.946 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:32.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-cephfs 2024-03-29T11:35:33.567 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:33.666 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:35:33.717 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:33.718 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-29T11:35:33.718 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:33.771 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rbd 2024-03-29T11:35:34.336 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:34.435 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:35:34.487 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:34.487 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-29T11:35:34.488 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:34.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-fuse 2024-03-29T11:35:35.107 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:35.260 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:35.261 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:35.261 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-29T11:35:35.261 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:35.261 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:35:35.261 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:35:35.262 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:35.262 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:35:35.262 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:35.262 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-29T11:35:35.262 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:35.262 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 85 k 2024-03-29T11:35:35.262 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 231 k 2024-03-29T11:35:35.262 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:35:35.473 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 409 kB/s | 85 kB 00:00 2024-03-29T11:35:35.474 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:35:35.474 INFO:teuthology.orchestra.run.smithi191.stdout:Total 406 kB/s | 85 kB 00:00 2024-03-29T11:35:35.474 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:35:35.478 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:35:35.478 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:35:35.533 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:35:35.533 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:35:35.666 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:35:35.785 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:36.177 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:36.477 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:36.477 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:36.477 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:35:36.477 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:35:36.477 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:36.477 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:36.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2024-03-29T11:35:37.314 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:37.475 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.1 M 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 13 M 2024-03-29T11:35:37.476 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:35:37.908 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 7.2 MB/s | 3.1 MB 00:00 2024-03-29T11:35:37.908 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:35:37.909 INFO:teuthology.orchestra.run.smithi191.stdout:Total 7.1 MB/s | 3.1 MB 00:00 2024-03-29T11:35:37.909 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:35:37.918 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:35:37.918 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:35:37.992 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:35:37.993 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:35:38.417 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:35:38.492 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:38.518 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:38.518 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:35:38.518 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:35:38.518 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:35:38.518 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:35:38.518 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:39.731 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:39.731 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:39.731 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:35:39.731 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:35:39.731 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:39.731 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:39.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-nbd 2024-03-29T11:35:40.661 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:33:52 AM UTC. 2024-03-29T11:35:40.810 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-29T11:35:40.810 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:40.810 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 171 k 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 494 k 2024-03-29T11:35:40.811 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-29T11:35:41.161 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 491 kB/s | 171 kB 00:00 2024-03-29T11:35:41.162 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:35:41.162 INFO:teuthology.orchestra.run.smithi191.stdout:Total 488 kB/s | 171 kB 00:00 2024-03-29T11:35:41.162 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-29T11:35:41.170 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-29T11:35:41.170 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-29T11:35:41.225 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-29T11:35:41.225 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-29T11:35:41.372 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-29T11:35:41.493 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:41.929 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:42.232 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:35:42.232 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:42.232 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-29T11:35:42.232 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:35:42.232 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:42.232 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-29T11:35:42.498 DEBUG:teuthology.parallel:result is None 2024-03-29T11:35:42.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:35:42.766 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:35:42.791 INFO:teuthology.orchestra.run.smithi100.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:35:42.792 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:35:42.792 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:35:42.793 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:35:43.007 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:35:43.028 INFO:teuthology.orchestra.run.smithi191.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:35:43.028 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:35:43.028 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:35:43.030 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:35:43.030 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:35:43.030 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:35:43.063 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:35:43.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:35:43.095 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:35:43.095 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:35:43.095 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:35:43.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:35:43.202 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:35:43.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:35:43.229 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:35:43.295 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:35:43.296 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:35:43.296 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:35:43.324 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:35:43.392 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:35:43.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:35:43.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:35:43.486 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:35:43.486 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:35:43.486 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:35:43.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:35:43.581 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:35:43.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:35:43.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:35:43.676 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:35:43.758 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:35:43.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 777 /var/log/ceph 2024-03-29T11:35:43.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2024-03-29T11:35:43.787 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:35:43.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:35:43.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:35:43.856 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:35:43.856 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:35:43.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:35:43.934 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:35:43.934 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:35:43.935 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622732', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi100.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5bf0bJYXI1z2YAYj2AirhKKyd1vhJ3GYcn3DfjFSYffBbXRw3uQpBRHkGxaMS3Jk2YN+vvtOZrRkqzX6g7kVY=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCXqkS0wvhnvppYL6bO+4o1t1ZrPFTGqFlyuLbR513BpWS3w1HfG9bfqSonOx7xNWAp6+8jBREvVeGStEZIeuuo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:35:43.935 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:35:43.965 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:35:43.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:35:44.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:35:44.047 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:35:44.048 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:35:44.099 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:35:44.099 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_1 2024-03-29T11:35:44.153 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:35:44.153 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:35:44.153 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:35:44.153 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:35:44.154 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:35:44.154 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-03-29 11:34:06.601814473 +0000 2024-03-29T11:35:44.154 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-03-29 11:30:50.923694458 +0000 2024-03-29T11:35:44.154 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-03-29 11:30:50.923694458 +0000 2024-03-29T11:35:44.154 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-03-29 11:30:50.923694458 +0000 2024-03-29T11:35:44.154 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:35:44.218 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-03-29T11:35:44.218 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-03-29T11:35:44.218 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.00018406 s, 2.8 MB/s 2024-03-29T11:35:44.219 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:35:44.274 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_2 2024-03-29T11:35:44.329 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:35:44.329 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:35:44.330 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:35:44.330 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:35:44.330 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:35:44.330 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-03-29 11:34:06.602814446 +0000 2024-03-29T11:35:44.330 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-03-29 11:30:51.270685186 +0000 2024-03-29T11:35:44.330 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-03-29 11:30:51.270685186 +0000 2024-03-29T11:35:44.330 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-03-29 11:30:51.270685186 +0000 2024-03-29T11:35:44.330 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:35:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-03-29T11:35:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-03-29T11:35:44.394 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000182025 s, 2.8 MB/s 2024-03-29T11:35:44.395 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:35:44.450 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_3 2024-03-29T11:35:44.505 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:35:44.505 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:35:44.505 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:35:44.505 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:35:44.505 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:35:44.505 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-03-29 11:34:06.602814446 +0000 2024-03-29T11:35:44.505 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-03-29 11:30:51.598676421 +0000 2024-03-29T11:35:44.506 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-03-29 11:30:51.598676421 +0000 2024-03-29T11:35:44.506 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-03-29 11:30:51.598676421 +0000 2024-03-29T11:35:44.506 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:35:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-03-29T11:35:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-03-29T11:35:44.572 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000168881 s, 3.0 MB/s 2024-03-29T11:35:44.573 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:35:44.628 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_4 2024-03-29T11:35:44.684 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:35:44.684 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:35:44.684 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:35:44.684 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:35:44.684 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:35:44.684 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-03-29 11:34:06.602814446 +0000 2024-03-29T11:35:44.684 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-03-29 11:30:51.919667844 +0000 2024-03-29T11:35:44.684 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-03-29 11:30:51.919667844 +0000 2024-03-29T11:35:44.684 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-03-29 11:30:51.919667844 +0000 2024-03-29T11:35:44.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:35:44.749 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-03-29T11:35:44.749 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-03-29T11:35:44.750 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000184729 s, 2.8 MB/s 2024-03-29T11:35:44.750 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:35:44.806 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:35:44.807 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:35:44.807 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:35:44.823 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:35:44.823 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-03-29T11:35:44.879 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:35:44.879 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:35:44.879 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:35:44.879 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:35:44.879 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:35:44.879 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-29 11:35:38.779561287 +0000 2024-03-29T11:35:44.880 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-29 11:30:50.934959299 +0000 2024-03-29T11:35:44.880 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-29 11:30:50.934959299 +0000 2024-03-29T11:35:44.880 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-29 11:30:50.934959299 +0000 2024-03-29T11:35:44.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:35:44.946 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-29T11:35:44.946 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-29T11:35:44.946 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000229373 s, 2.2 MB/s 2024-03-29T11:35:44.947 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:35:45.005 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-03-29T11:35:45.061 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:35:45.061 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:35:45.061 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:35:45.061 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:35:45.062 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:35:45.062 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-29 11:35:38.779561287 +0000 2024-03-29T11:35:45.062 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-29 11:30:51.267953340 +0000 2024-03-29T11:35:45.062 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-29 11:30:51.267953340 +0000 2024-03-29T11:35:45.062 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-29 11:30:51.267953340 +0000 2024-03-29T11:35:45.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:35:45.132 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-29T11:35:45.132 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-29T11:35:45.133 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000193447 s, 2.6 MB/s 2024-03-29T11:35:45.133 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:35:45.190 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-03-29T11:35:45.246 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:35:45.247 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:35:45.247 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:35:45.247 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:35:45.247 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:35:45.247 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-29 11:35:38.780561269 +0000 2024-03-29T11:35:45.247 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-29 11:30:51.605947290 +0000 2024-03-29T11:35:45.247 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-29 11:30:51.605947290 +0000 2024-03-29T11:35:45.247 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-29 11:30:51.605947290 +0000 2024-03-29T11:35:45.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:35:45.313 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-29T11:35:45.313 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-29T11:35:45.313 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00023602 s, 2.2 MB/s 2024-03-29T11:35:45.314 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:35:45.371 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-03-29T11:35:45.426 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:35:45.426 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:35:45.427 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:35:45.427 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:35:45.427 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:35:45.427 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-29 11:35:38.780561269 +0000 2024-03-29T11:35:45.427 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-29 11:30:51.926941545 +0000 2024-03-29T11:35:45.427 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-29 11:30:51.926941545 +0000 2024-03-29T11:35:45.427 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-29 11:30:51.926941545 +0000 2024-03-29T11:35:45.427 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:35:45.493 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-29T11:35:45.493 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-29T11:35:45.493 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000238249 s, 2.1 MB/s 2024-03-29T11:35:45.494 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:35:45.553 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:35:45.553 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:35:45.553 INFO:tasks.ceph:Generating config... 2024-03-29T11:35:45.555 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:35:45.555 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:35:45.555 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:35:45.555 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:35:45.555 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:35:45.555 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:35:45.555 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:35:45.555 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:35:45.555 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:35:45.555 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:35:45.555 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:35:45.555 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:35:45.555 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:35:45.555 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:35:45.556 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:35:45.556 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:35:45.556 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:35:45.556 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:35:45.556 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:35:45.556 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:35:45.556 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:35:45.556 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:35:45.556 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:35:45.556 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:35:45.556 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:35:45.556 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:35:45.556 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:35:45.556 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:35:45.556 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:35:45.557 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:35:45.558 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:35:45.558 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:35:45.558 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:35:45.558 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:35:45.558 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:35:45.558 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:35:45.617 INFO:teuthology.orchestra.run.smithi100.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:35:45.620 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:35:45.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:35:45.693 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.100:6789]'), ('mon.c', '[v1:172.21.15.100:6790]'), ('mon.b', '[v1:172.21.15.191:6789]')] 2024-03-29T11:35:45.694 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.100:6789],[v1:172.21.15.100:6790],[v1:172.21.15.191:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:35:45.695 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:35:45.695 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:35:45.749 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.100:6789]' --addv c '[v1:172.21.15.100:6790]' --addv b '[v1:172.21.15.191:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:35:45.834 INFO:teuthology.orchestra.run.smithi100.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:monmaptool: generated fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:setting min_mon_release = quincy 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:epoch 0 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:min_mon_release 17 (quincy) 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:election_strategy: 1 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:35:45.835 INFO:teuthology.orchestra.run.smithi100.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:35:45.836 DEBUG:teuthology.orchestra.run.smithi100:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:35:45.851 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fd8785f3-b448-4d0b-b3ce-b69d1146f339... 2024-03-29T11:35:45.852 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:35:45.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:35:45.941 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-03-29T11:35:45.941 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-03-29T11:35:45.941 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:35:45.941 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-03-29T11:35:45.941 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:35:45.942 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:35:45.943 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "[v1:172.21.15.100:6789],[v1:172.21.15.100:6790],[v1:172.21.15.191:6789]" 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: lockdep = True 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: mon client directed command retry = 5 2024-03-29T11:35:45.944 INFO:teuthology.orchestra.run.smithi100.stdout: mon election default strategy = 1 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: ms bind msgr2 = False 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: ms inject socket failures = 5000 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: ms type = async 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.945 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug trim objects = true 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-03-29T11:35:45.946 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore compression algorithm = snappy 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore compression mode = aggressive 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore zero block detection = True 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 20 2024-03-29T11:35:45.947 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 20 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: debug monc = 20 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 10 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: osd class default list = * 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: osd class load list = * 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: osd client watch timeout = 120 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify cached snaps = True 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify missing on start = True 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:35:45.948 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock override recovery settings = True 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: osd op queue = debug_random 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: osd op queue cut off = debug_random 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: osd shutdown pgref assert = True 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.949 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-03-29T11:35:45.950 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-03-29T11:35:45.950 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-03-29T11:35:45.950 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-03-29T11:35:45.950 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-03-29T11:35:45.950 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-03-29T11:35:45.950 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-03-29T11:35:45.950 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:35:45.950 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-03-29T11:35:45.950 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-03-29T11:35:45.950 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:35:45.951 INFO:teuthology.orchestra.run.smithi100.stdout: mon min osdmap epochs = 50 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune interval = 2 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune min = 15 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: mon scrub interval = 300 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on pool no app = False 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: paxos service trim min = 10 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: debug monc = 20 2024-03-29T11:35:45.952 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-03-29T11:35:45.953 INFO:teuthology.orchestra.run.smithi100.stdout: debug objecter = 20 2024-03-29T11:35:45.953 INFO:teuthology.orchestra.run.smithi100.stdout: debug rados = 20 2024-03-29T11:35:45.953 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-03-29T11:35:45.953 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-03-29T11:35:45.953 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-03-29T11:35:45.953 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2024-03-29T11:35:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2024-03-29T11:35:45.953 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2024-03-29T11:35:45.954 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2024-03-29T11:35:45.955 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2024-03-29T11:35:45.956 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "[v1:172.21.15.100:6789],[v1:172.21.15.100:6790],[v1:172.21.15.191:6789]" 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: lockdep = True 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: mon election default strategy = 1 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: ms bind msgr2 = False 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 5000 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: ms type = async 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2024-03-29T11:35:45.957 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug trim objects = true 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2024-03-29T11:35:45.958 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression algorithm = snappy 2024-03-29T11:35:45.959 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression mode = aggressive 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore zero block detection = True 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 20 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 20 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: debug monc = 20 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 10 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: osd class default list = * 2024-03-29T11:35:45.960 INFO:teuthology.orchestra.run.smithi191.stdout: osd class load list = * 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: osd client watch timeout = 120 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify cached snaps = True 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify missing on start = True 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock override recovery settings = True 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: osd op queue = debug_random 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: osd op queue cut off = debug_random 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: osd shutdown pgref assert = True 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-03-29T11:35:45.961 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:35:45.962 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.963 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: mon min osdmap epochs = 50 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune interval = 2 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune min = 15 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: mon scrub interval = 300 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on pool no app = False 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: paxos service trim min = 10 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:35:45.964 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:35:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: debug monc = 20 2024-03-29T11:35:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-29T11:35:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: debug objecter = 20 2024-03-29T11:35:45.965 INFO:teuthology.orchestra.run.smithi191.stdout: debug rados = 20 2024-03-29T11:35:45.965 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2024-03-29T11:35:45.965 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2024-03-29T11:35:45.965 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2024-03-29T11:35:45.968 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:35:45.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:35:46.023 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:35:46.023 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:35:46.023 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:35:46.038 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:35:46.038 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:35:46.093 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:35:46.093 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:35:46.093 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:35:46.093 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:35:46.173 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:35:46.173 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:35:46.228 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:35:46.228 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:35:46.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:35:46.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:35:46.268 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:35:46.268 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:35:46.323 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:35:46.323 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:35:46.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:35:46.377 INFO:teuthology.orchestra.run.smithi100.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:35:46.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:35:46.461 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:35:46.464 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:35:46.464 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:35:46.464 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:35:46.506 INFO:teuthology.orchestra.run.smithi100.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:35:46.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:35:46.563 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:35:46.578 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:35:46.579 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:35:46.579 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:35:46.605 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:35:46.606 INFO:tasks.ceph:role: osd.0 2024-03-29T11:35:46.606 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:35:46.606 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:35:46.680 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:35:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:35:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:35:46.680 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:35:46.680 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:35:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:35:46.681 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:35:46.681 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:35:46.681 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:35:46.681 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:35:47.187 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-03-29T11:35:47.189 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-03-29T11:35:47.189 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:35:47.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:35:47.360 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:35:47.428 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:35:47.428 INFO:tasks.ceph:role: osd.1 2024-03-29T11:35:47.428 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:35:47.429 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:35:47.503 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:35:47.503 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:35:47.504 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:35:47.504 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:35:47.504 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:35:47.504 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:35:47.504 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:35:47.504 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:35:47.504 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:35:47.504 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:35:47.963 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-03-29T11:35:47.965 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-03-29T11:35:47.965 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:35:48.061 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:35:48.094 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:35:48.162 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:35:48.162 INFO:tasks.ceph:role: osd.2 2024-03-29T11:35:48.163 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:35:48.163 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:35:48.237 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:35:48.238 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:35:48.238 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:35:48.238 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:35:48.238 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:35:48.238 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:35:48.238 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:35:48.238 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:35:48.238 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:35:48.238 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:35:48.729 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-03-29T11:35:48.731 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-03-29T11:35:48.731 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:35:48.881 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:35:48.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:35:48.986 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:35:48.986 INFO:tasks.ceph:role: osd.3 2024-03-29T11:35:48.986 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:35:48.986 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:35:49.062 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:35:49.062 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:35:49.062 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:35:49.062 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:35:49.062 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:35:49.063 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:35:49.063 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:35:49.063 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:35:49.063 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:35:49.063 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:35:49.568 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-03-29T11:35:49.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-03-29T11:35:49.571 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:35:49.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:35:49.745 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:35:49.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:49.846+0000 7fe4f640e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:35:49.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:49.846+0000 7fe4f640e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:35:49.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:49.846+0000 7fe4f640e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:35:49.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:49.846+0000 7fe4f640e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:35:49.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:49.846+0000 7fe4f640e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:35:49.845 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:49.847+0000 7fe4f640e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:35:52.693 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:35:52.725 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:35:52.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:52.817+0000 7f647c5df640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:35:52.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:52.817+0000 7f647c5df640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:35:52.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:52.817+0000 7f647c5df640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:35:52.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:52.817+0000 7f647c5df640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:35:52.816 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:52.817+0000 7f647c5df640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:35:52.817 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:52.818+0000 7f647c5df640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:35:55.662 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:35:55.692 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:35:55.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:55.783+0000 7f655fa1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:35:55.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:55.783+0000 7f655fa1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:35:55.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:55.783+0000 7f655fa1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:35:55.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:55.783+0000 7f655fa1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:35:55.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:55.783+0000 7f655fa1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:35:55.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:55.784+0000 7f655fa1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:35:58.649 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:35:58.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:35:58.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:58.772+0000 7ff8104a4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:35:58.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:58.772+0000 7ff8104a4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:35:58.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:58.772+0000 7ff8104a4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:35:58.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:58.772+0000 7ff8104a4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:35:58.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:58.772+0000 7ff8104a4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:35:58.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:35:58.773+0000 7ff8104a4640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:36:01.620 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:36:01.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:36:01.682 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:36:01.682 INFO:tasks.ceph:role: osd.4 2024-03-29T11:36:01.682 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:36:01.682 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:36:01.760 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:36:01.760 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:36:01.760 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:36:01.760 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:36:01.760 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:36:01.760 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:36:01.760 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:36:01.761 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:36:01.761 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:36:01.761 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:36:02.239 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-29T11:36:02.241 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-29T11:36:02.241 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:36:02.366 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:36:02.402 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:36:02.472 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:36:02.472 INFO:tasks.ceph:role: osd.5 2024-03-29T11:36:02.472 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:36:02.472 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:36:02.548 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:36:02.548 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:36:02.548 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:36:02.548 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:36:02.548 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:36:02.549 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:36:02.549 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:36:02.549 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:36:02.549 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:36:02.549 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:36:03.006 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-29T11:36:03.008 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-29T11:36:03.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:36:03.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:36:03.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:36:03.197 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:36:03.197 INFO:tasks.ceph:role: osd.6 2024-03-29T11:36:03.198 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:36:03.198 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:36:03.271 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:36:03.272 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:36:03.272 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:36:03.272 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:36:03.272 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:36:03.272 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:36:03.272 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:36:03.272 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:36:03.272 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:36:03.272 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:36:03.756 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-29T11:36:03.757 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-29T11:36:03.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:36:03.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:36:03.931 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:36:03.998 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:36:03.998 INFO:tasks.ceph:role: osd.7 2024-03-29T11:36:03.998 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:36:03.998 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:36:04.072 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:36:04.072 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:36:04.072 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:36:04.072 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:36:04.072 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:36:04.072 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:36:04.072 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:36:04.072 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:36:04.072 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:36:04.072 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:36:04.559 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-29T11:36:04.561 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-29T11:36:04.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:36:04.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:36:04.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:36:04.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:04.839+0000 7fbddcc19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:36:04.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:04.839+0000 7fbddcc19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:36:04.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:04.839+0000 7fbddcc19640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:36:04.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:04.839+0000 7fbddcc19640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:36:04.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:04.839+0000 7fbddcc19640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:36:04.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:04.840+0000 7fbddcc19640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:36:07.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:36:07.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:36:07.812 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:07.814+0000 7fcc1fc19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:36:07.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:07.814+0000 7fcc1fc19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:36:07.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:07.814+0000 7fcc1fc19640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:36:07.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:07.814+0000 7fcc1fc19640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:36:07.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:07.814+0000 7fcc1fc19640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:36:07.813 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:07.814+0000 7fcc1fc19640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:36:10.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:36:10.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:36:10.778 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:10.778+0000 7fae69b36640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:36:10.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:10.778+0000 7fae69b36640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:36:10.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:10.778+0000 7fae69b36640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:36:10.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:10.778+0000 7fae69b36640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:36:10.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:10.778+0000 7fae69b36640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:36:10.779 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:10.779+0000 7fae69b36640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:36:13.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:36:13.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:36:13.755 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:13.755+0000 7f56cb287640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:36:13.755 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:13.755+0000 7f56cb287640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:36:13.755 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:13.755+0000 7f56cb287640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:36:13.755 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:13.755+0000 7f56cb287640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:36:13.755 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:13.755+0000 7f56cb287640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:36:13.756 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:13.756+0000 7f56cb287640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:36:16.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:36:16.645 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:36:16.645 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:36:16.646 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:36:16.678 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:36:16.678 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:36:16.748 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:36:16.748 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:36:16.817 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:36:16.817 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:36:16.887 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:36:16.887 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:36:16.955 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:36:16.955 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:36:16.986 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:36:16.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:36:17.057 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:36:17.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:36:17.127 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:36:17.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:36:17.196 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:36:17.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:36:17.264 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:36:17.264 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:36:17.281 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:36:17.281 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:36:17.320 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:36:17.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:36:17.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:36:17.364 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.y] 2024-03-29T11:36:17.364 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCSpwZmS3WsFhAAoTGnuhDlHMoI/liVtrZqtQ== 2024-03-29T11:36:17.364 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.0] 2024-03-29T11:36:17.364 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCVpwZmm/qBMhAAAMraVZB4liFda5mjAx4R3g== 2024-03-29T11:36:17.364 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.1] 2024-03-29T11:36:17.364 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCYpwZmFYrIMBAATyF+nhLQ10BSJbYaCoFMYA== 2024-03-29T11:36:17.364 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.2] 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCbpwZmHN/ALhAA2qeeQduIRs9mJdLcd9MJfw== 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.3] 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCepwZmQjoXLhAA3sPQA7aLkwRaZyO4xPDHzg== 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.x] 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCSpwZme5qzGxAATNzO/OamzCt/cPjhl3pWMQ== 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.4] 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCkpwZmuDEVMhAANaz/LjJIAo1UipjS+yiTAQ== 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.5] 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCnpwZmO6CUMBAA5suIaFjilOktioUIUygIuw== 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.6] 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCqpwZmH7ODLhAA7hlmh2jebKLmvFDgDQrkdQ== 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.7] 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCtpwZm27kkLRAACl27AmDS51EPYYW4ZRDG0Q== 2024-03-29T11:36:17.365 INFO:teuthology.orchestra.run.smithi100.stdout:[client.0] 2024-03-29T11:36:17.366 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCSpwZmBQJZHhAAhxJF9yZlwNTb05/ru6waXg== 2024-03-29T11:36:17.366 INFO:teuthology.orchestra.run.smithi100.stdout:[client.1] 2024-03-29T11:36:17.366 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCSpwZmjdjEIRAAtExxneJsTdPiVQePqNjzoQ== 2024-03-29T11:36:17.386 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.y] 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCSpwZmS3WsFhAAoTGnuhDlHMoI/liVtrZqtQ== 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCVpwZmm/qBMhAAAMraVZB4liFda5mjAx4R3g== 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCYpwZmFYrIMBAATyF+nhLQ10BSJbYaCoFMYA== 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCbpwZmHN/ALhAA2qeeQduIRs9mJdLcd9MJfw== 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.3] 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCepwZmQjoXLhAA3sPQA7aLkwRaZyO4xPDHzg== 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCSpwZme5qzGxAATNzO/OamzCt/cPjhl3pWMQ== 2024-03-29T11:36:17.387 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.4] 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCkpwZmuDEVMhAANaz/LjJIAo1UipjS+yiTAQ== 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.5] 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCnpwZmO6CUMBAA5suIaFjilOktioUIUygIuw== 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.6] 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCqpwZmH7ODLhAA7hlmh2jebKLmvFDgDQrkdQ== 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.7] 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCtpwZm27kkLRAACl27AmDS51EPYYW4ZRDG0Q== 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCSpwZmBQJZHhAAhxJF9yZlwNTb05/ru6waXg== 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout:[client.1] 2024-03-29T11:36:17.388 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCSpwZmjdjEIRAAtExxneJsTdPiVQePqNjzoQ== 2024-03-29T11:36:17.389 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:36:17.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:36:17.482 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.536 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.590 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.644 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.699 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:36:17.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:36:17.753 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.755 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.807 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.914 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:36:17.968 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:36:17.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:36:18.026 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:36:18.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:36:18.083 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:36:18.083 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:36:18.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:36:18.206 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:36:18.233 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:36:18.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:36:18.401 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:36:18.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:36:18.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:36:18.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:36:18.578 DEBUG:teuthology.orchestra.run.smithi100:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:36:18.580 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:36:18.632 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:36:18.632 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:36:18.633 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:36:18.634 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:36:18.635 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:36:18.635 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:36:18.637 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:36:18.637 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:36:18.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:36:18.675 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:36:18.675 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:36:18.675 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:36:18.675 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:36:18.677 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:36:18.678 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:36:18.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:36:18.679 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:36:18.680 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:36:18.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:36:18.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.782+0000 7f574b13b640 10 monclient: get_monmap_and_config 2024-03-29T11:36:18.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.782+0000 7f574b13b640 10 monclient: build_initial_monmap 2024-03-29T11:36:18.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.782+0000 7f574b13b640 10 monclient: monmap: 2024-03-29T11:36:18.781 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:36:18.783164+0000 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:36:18.783164+0000 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.782+0000 7f574b13b640 1 Processor -- start 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.782+0000 7f574b13b640 1 -- start start 2024-03-29T11:36:18.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.782+0000 7f574b13b640 10 monclient: init 2024-03-29T11:36:18.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f574b13b640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:18.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f574b13b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:18.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f574b13b640 10 monclient(hunting): picked mon.noname-b con 0x7f574409ac40 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:18.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f574b13b640 10 monclient(hunting): picked mon.noname-c con 0x7f574409c870 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:18.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f574b13b640 10 monclient(hunting): picked mon.noname-a con 0x7f57440b4360 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:18.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f574b13b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574409dec0 con 0x7f57440b4360 2024-03-29T11:36:18.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f574b13b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574409da30 con 0x7f574409c870 2024-03-29T11:36:18.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f574b13b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57440b39c0 con 0x7f574409ac40 2024-03-29T11:36:18.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f574b13b640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:18.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f574b13b640 10 monclient(hunting): authenticate will time out at 898.978636s 2024-03-29T11:36:18.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f5748ed7640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f574409ac40 0x7f57440b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:37256/0 (socket says 172.21.15.100:37256) 2024-03-29T11:36:18.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:18.783+0000 7f5748ed7640 1 -- 172.21.15.100:0/1430769055 learned_addr learned my addr 172.21.15.100:0/1430769055 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 10 monclient(hunting): tick 2024-03-29T11:36:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:36:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 1 monclient(hunting): continuing hunt 2024-03-29T11:36:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:36:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 1 -- 172.21.15.100:0/1430769055 >> v1:172.21.15.100:6790/0 conn(0x7f574409ac40 legacy=0x7f57440b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 1 -- 172.21.15.100:0/1430769055 >> v1:172.21.15.191:6789/0 conn(0x7f574409c870 legacy=0x7f574413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 1 -- 172.21.15.100:0/1430769055 >> v1:172.21.15.100:6789/0 conn(0x7f57440b4360 legacy=0x7f574414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:21.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 10 monclient(hunting): picked mon.noname-c con 0x7f57280036e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:21.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 10 monclient(hunting): picked mon.noname-b con 0x7f574409ac40 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:21.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 10 monclient(hunting): picked mon.noname-a con 0x7f572800ab20 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:21.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 1 -- 172.21.15.100:0/1430769055 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57440b5160 con 0x7f572800ab20 2024-03-29T11:36:21.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 1 -- 172.21.15.100:0/1430769055 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572800f7f0 con 0x7f57280036e0 2024-03-29T11:36:21.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 1 -- 172.21.15.100:0/1430769055 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5728010a50 con 0x7f574409ac40 2024-03-29T11:36:21.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:21.784+0000 7f57427fc640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:23.948 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615177917 0 0) 0x7f57440b5160 con 0x7f572800ab20 2024-03-29T11:36:23.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:36:23.949 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5734003520 con 0x7f572800ab20 2024-03-29T11:36:23.951 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54688346 0 0) 0x7f572800f7f0 con 0x7f57280036e0 2024-03-29T11:36:23.951 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.951 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57440b5160 con 0x7f57280036e0 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 263445291 0 0) 0x7f57440b5160 con 0x7f57280036e0 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 >> v1:172.21.15.100:6790/0 conn(0x7f574409ac40 legacy=0x7f5728009320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.950+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 >> v1:172.21.15.100:6789/0 conn(0x7f572800ab20 legacy=0x7f572800cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 1 monclient: found mon.noname-c 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5728011cb0 con 0x7f57280036e0 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:23.952 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5738001610 con 0x7f57280036e0 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: dump: 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:23.953 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f574b13b640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f574b13b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5738001770 con 0x7f57280036e0 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 1 -- 172.21.15.100:0/1430769055 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5738001b80 con 0x7f57280036e0 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f574b13b640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f574b13b640 4 set_mon_vals no callback set 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.951+0000 7f57437fe640 10 monclient: dump: 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:23.954 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.952+0000 7f574b13b640 10 monclient: shutdown 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.952+0000 7f574b13b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.952+0000 7f574b13b640 1 -- 172.21.15.100:0/1430769055 >> v1:172.21.15.191:6789/0 conn(0x7f57280036e0 legacy=0x7f5728005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.952+0000 7f574b13b640 1 -- 172.21.15.100:0/1430769055 shutdown_connections 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.952+0000 7f574b13b640 1 -- 172.21.15.100:0/1430769055 >> 172.21.15.100:0/1430769055 conn(0x7f5744066b00 msgr2=0x7f5744066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.952+0000 7f574b13b640 1 -- 172.21.15.100:0/1430769055 shutdown_connections 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.952+0000 7f574b13b640 1 -- 172.21.15.100:0/1430769055 wait complete. 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.953+0000 7f574b13b640 10 monclient: build_initial_monmap 2024-03-29T11:36:23.955 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.953+0000 7f574b13b640 10 monclient: monmap: 2024-03-29T11:36:23.956 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-03-29T11:36:23.956 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:23.956 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-03-29T11:36:23.956 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-03-29T11:36:23.956 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:23.956 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:23.956 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:23.956 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:23.956 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:23.956 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:23.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.953+0000 7f574b13b640 1 librados: starting msgr at 2024-03-29T11:36:23.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.953+0000 7f574b13b640 1 librados: starting objecter 2024-03-29T11:36:23.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.953+0000 7f574b13b640 1 Processor -- start 2024-03-29T11:36:23.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.954+0000 7f574b13b640 1 -- start start 2024-03-29T11:36:23.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.954+0000 7f574b13b640 1 librados: setting wanted keys 2024-03-29T11:36:23.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.954+0000 7f574b13b640 1 librados: calling monclient init 2024-03-29T11:36:23.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.954+0000 7f574b13b640 10 monclient: init 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.954+0000 7f574b13b640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.954+0000 7f574b13b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.954+0000 7f574b13b640 10 monclient(hunting): picked mon.noname-c con 0x7f57280036e0 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f574b13b640 10 monclient(hunting): picked mon.noname-b con 0x7f572800ab20 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f574b13b640 10 monclient(hunting): picked mon.noname-a con 0x7f574409ac40 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f574b13b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5744143530 con 0x7f574409ac40 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f574b13b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5744143780 con 0x7f572800ab20 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f574b13b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5744143a60 con 0x7f57280036e0 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f574b13b640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f574b13b640 10 monclient(hunting): authenticate will time out at 904.150197s 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f5748ed7640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f57280036e0 0x7f5744131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:54956/0 (socket says 172.21.15.100:54956) 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f5748ed7640 1 -- 172.21.15.100:0/454157808 learned_addr learned my addr 172.21.15.100:0/454157808 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f5748ed7640 1 -- 172.21.15.100:0/454157808 >> v1:172.21.15.100:6790/0 conn(0x7f57280036e0 legacy=0x7f5744131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:36:23.961 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f5748ed7640 1 -- 172.21.15.100:0/454157808 >> v1:172.21.15.100:6790/0 conn(0x7f57280036e0 legacy=0x7f5744131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f5748ed7640 1 --1- 172.21.15.100:0/454157808 >> v1:172.21.15.100:6790/0 conn(0x7f57280036e0 0x7f5744131d30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f5748ed7640 1 --1- 172.21.15.100:0/454157808 >> v1:172.21.15.100:6790/0 conn(0x7f57280036e0 0x7f5744131d30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.100:6790/0 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 client.?.objecter ms_handle_connect 0x7f574409ac40 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 client.?.objecter ms_handle_connect 0x7f572800ab20 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723901107 0 0) 0x7f5744143530 con 0x7f574409ac40 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57280059f0 con 0x7f574409ac40 2024-03-29T11:36:23.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276437359 0 0) 0x7f57280059f0 con 0x7f574409ac40 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5744143530 con 0x7f574409ac40 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f573c001a50 con 0x7f574409ac40 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.955+0000 7f57417fa640 10 monclient(hunting): dump: 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:23.963 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 446001642 0 0) 0x7f5744143530 con 0x7f574409ac40 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 >> v1:172.21.15.100:6790/0 conn(0x7f57280036e0 legacy=0x7f5744131d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 >> v1:172.21.15.191:6789/0 conn(0x7f572800ab20 legacy=0x7f5744140030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 1 monclient: found mon.a 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5744142560 con 0x7f574409ac40 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 10 monclient: _finish_auth 0 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f574b13b640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f574b13b640 10 monclient: _renew_subs 2024-03-29T11:36:23.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f574b13b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f574b13b640 1 -- 172.21.15.100:0/454157808 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5744136ed0 con 0x7f574409ac40 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f574b13b640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f574b13b640 10 monclient: _renew_subs 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f574b13b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f574b13b640 1 -- 172.21.15.100:0/454157808 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5744132440 con 0x7f574409ac40 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f574b13b640 1 librados: init done 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f573c001ee0 con 0x7f574409ac40 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f573c003b10 con 0x7f574409ac40 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 10 monclient: dump: 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-03-29T11:36:23.965 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f5742ffd640 4 set_mon_vals no callback set 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f573c004000 con 0x7f574409ac40 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2255356038 0 0) 0x7f573c005490 con 0x7f574409ac40 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 10 client.4110.objecter ms_dispatch 0x7f5744000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:36:23.966 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:36:23.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.956+0000 7f57417fa640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:36:23.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.957+0000 7f574b13b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:23.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.957+0000 7f574b13b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:23.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.957+0000 7f574b13b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:23.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.957+0000 7f574b13b640 1 -- 172.21.15.100:0/454157808 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5724005080 con 0x7f574409ac40 2024-03-29T11:36:23.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.961+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f573c0056c0 con 0x7f574409ac40 2024-03-29T11:36:23.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.961+0000 7f57417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:23.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:23.961+0000 7f57417fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:24.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.051+0000 7f574b13b640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:36:24.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.051+0000 7f574b13b640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:36:24.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.051+0000 7f574b13b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:24.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.051+0000 7f574b13b640 1 -- 172.21.15.100:0/454157808 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5724005330 con 0x7f574409ac40 2024-03-29T11:36:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.053+0000 7f57417fa640 1 -- 172.21.15.100:0/454157808 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f573c004480 con 0x7f574409ac40 2024-03-29T11:36:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.053+0000 7f57417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:36:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.053+0000 7f57417fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:36:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:adjusted tunables profile to default 2024-03-29T11:36:24.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.053+0000 7f574b13b640 10 librados: watch_flush enter 2024-03-29T11:36:24.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.053+0000 7f574b13b640 10 librados: watch_flush exit 2024-03-29T11:36:24.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.053+0000 7f574b13b640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:36:24.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.053+0000 7f574b13b640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:36:24.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.053+0000 7f574b13b640 10 monclient: shutdown 2024-03-29T11:36:24.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.053+0000 7f574b13b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:24.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.053+0000 7f574b13b640 1 -- 172.21.15.100:0/454157808 >> v1:172.21.15.100:6789/0 conn(0x7f574409ac40 legacy=0x7f5744140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.054+0000 7f574b13b640 1 -- 172.21.15.100:0/454157808 shutdown_connections 2024-03-29T11:36:24.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.054+0000 7f574b13b640 1 -- 172.21.15.100:0/454157808 >> 172.21.15.100:0/454157808 conn(0x7f5744066b00 msgr2=0x7f5744066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:24.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.054+0000 7f574b13b640 1 -- 172.21.15.100:0/454157808 shutdown_connections 2024-03-29T11:36:24.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.054+0000 7f574b13b640 1 -- 172.21.15.100:0/454157808 wait complete. 2024-03-29T11:36:24.054 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:24.054+0000 7f574b13b640 1 librados: shutdown 2024-03-29T11:36:24.063 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:36:24.063 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:36:24.063 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:36:24.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:36:24.091 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:36:24.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:36:24.157 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:36:24.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:36:24.224 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-29T11:36:24.224 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:36:24.291 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:36:24.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:36:24.318 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:36:24.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:36:24.386 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:36:24.386 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:36:24.452 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-29T11:36:24.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:36:24.453 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:24.454+0000 7fda32e32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:36:24.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new bafc88bc-7772-46af-81dc-9ab1f21e7bbd 0 2024-03-29T11:36:24.480 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:24.481+0000 7f1b76fdb040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:36:24.509 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:24.511+0000 7fda32e32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:36:24.535 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:24.535+0000 7f1b76fdb040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.609+0000 7efe8c796640 10 monclient: get_monmap_and_config 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.609+0000 7efe8c796640 10 monclient: build_initial_monmap 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.609+0000 7efe8c796640 10 monclient: monmap: 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:36:24.610885+0000 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:36:24.610885+0000 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:24.609 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:24.610 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:24.610 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.610 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.610+0000 7efe8c796640 1 Processor -- start 2024-03-29T11:36:24.610 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.610+0000 7efe8c796640 1 -- start start 2024-03-29T11:36:24.610 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.610+0000 7efe8c796640 10 monclient: init 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 10 monclient(hunting): picked mon.noname-c con 0x7efe8409ac40 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 10 monclient(hunting): picked mon.noname-b con 0x7efe8409c870 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 10 monclient(hunting): picked mon.noname-a con 0x7efe840b4360 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8409dec0 con 0x7efe840b4360 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8409da30 con 0x7efe8409ac40 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe840b39c0 con 0x7efe8409c870 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 10 monclient(hunting): authenticate will time out at 894.928426s 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8a532640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7efe8409ac40 0x7efe840b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49832/0 (socket says 172.21.15.191:49832) 2024-03-29T11:36:24.619 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8a532640 1 -- 172.21.15.191:0/2118531351 learned_addr learned my addr 172.21.15.191:0/2118531351 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217394067 0 0) 0x7efe8409da30 con 0x7efe8409ac40 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe70003520 con 0x7efe8409ac40 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975568910 0 0) 0x7efe840b39c0 con 0x7efe8409c870 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe8409da30 con 0x7efe8409c870 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723083141 0 0) 0x7efe7c001490 con 0x7efe840b4360 2024-03-29T11:36:24.620 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe840b39c0 con 0x7efe840b4360 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1982506977 0 0) 0x7efe70003520 con 0x7efe8409ac40 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 >> v1:172.21.15.100:6790/0 conn(0x7efe8409c870 legacy=0x7efe8413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 >> v1:172.21.15.100:6789/0 conn(0x7efe840b4360 legacy=0x7efe8414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient: _finish_hunting 0 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 monclient: found mon.noname-c 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe840b5160 con 0x7efe8409ac40 2024-03-29T11:36:24.621 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient: _finish_auth 0 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient: _check_auth_tickets 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7efe74001690 con 0x7efe8409ac40 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe89530640 10 monclient: dump: 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.611+0000 7efe8c796640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe74001870 con 0x7efe8409ac40 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe89530640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe89530640 1 -- 172.21.15.191:0/2118531351 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7efe74001c80 con 0x7efe8409ac40 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe8c796640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe8c796640 4 set_mon_vals no callback set 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe89530640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe89530640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe89530640 10 monclient: dump: 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.623 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe8c796640 10 monclient: shutdown 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe8c796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe8c796640 1 -- 172.21.15.191:0/2118531351 >> v1:172.21.15.191:6789/0 conn(0x7efe8409ac40 legacy=0x7efe840b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe8c796640 1 -- 172.21.15.191:0/2118531351 shutdown_connections 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe8c796640 1 -- 172.21.15.191:0/2118531351 >> 172.21.15.191:0/2118531351 conn(0x7efe84066b00 msgr2=0x7efe84066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe8c796640 1 -- 172.21.15.191:0/2118531351 shutdown_connections 2024-03-29T11:36:24.624 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.612+0000 7efe8c796640 1 -- 172.21.15.191:0/2118531351 wait complete. 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.613+0000 7efe8c796640 10 monclient: build_initial_monmap 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.613+0000 7efe8c796640 10 monclient: monmap: 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.614+0000 7efe8c796640 1 librados: starting msgr at 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.614+0000 7efe8c796640 1 librados: starting objecter 2024-03-29T11:36:24.625 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.614+0000 7efe8c796640 1 Processor -- start 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.614+0000 7efe8c796640 1 -- start start 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.614+0000 7efe8c796640 1 librados: setting wanted keys 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.614+0000 7efe8c796640 1 librados: calling monclient init 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.614+0000 7efe8c796640 10 monclient: init 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8c796640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8c796640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8c796640 10 monclient(hunting): picked mon.noname-b con 0x7efe8409ac40 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8c796640 10 monclient(hunting): picked mon.noname-c con 0x7efe840af1d0 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8c796640 10 monclient(hunting): picked mon.noname-a con 0x7efe84139370 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8c796640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe841d4be0 con 0x7efe84139370 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8c796640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe840b06a0 con 0x7efe8409ac40 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8c796640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe840b1910 con 0x7efe840af1d0 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8c796640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:24.626 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8c796640 10 monclient(hunting): authenticate will time out at 894.932494s 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8a532640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7efe8409ac40 0x7efe840afdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49834/0 (socket says 172.21.15.191:49834) 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe8a532640 1 -- 172.21.15.191:0/1018249811 learned_addr learned my addr 172.21.15.191:0/1018249811 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 client.?.objecter ms_handle_connect 0x7efe8409ac40 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485186569 0 0) 0x7efe840b06a0 con 0x7efe8409ac40 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe68004eb0 con 0x7efe8409ac40 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 client.?.objecter ms_handle_connect 0x7efe840af1d0 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 client.?.objecter ms_handle_connect 0x7efe84139370 2024-03-29T11:36:24.627 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082917321 0 0) 0x7efe841d4be0 con 0x7efe84139370 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe840b06a0 con 0x7efe84139370 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938772219 0 0) 0x7efe840b1910 con 0x7efe840af1d0 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe841d4be0 con 0x7efe840af1d0 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4089300321 0 0) 0x7efe68004eb0 con 0x7efe8409ac40 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe840b1910 con 0x7efe8409ac40 2024-03-29T11:36:24.628 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7efe74001bc0 con 0x7efe8409ac40 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): dump: 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.629 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.615+0000 7efe7affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956671322 0 0) 0x7efe840b06a0 con 0x7efe84139370 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe68004eb0 con 0x7efe84139370 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7efe7c001b40 con 0x7efe84139370 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient(hunting): dump: 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.630 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1613718736 0 0) 0x7efe841d4be0 con 0x7efe840af1d0 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe840b06a0 con 0x7efe840af1d0 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7efe80001790 con 0x7efe840af1d0 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient(hunting): dump: 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.631 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2298588368 0 0) 0x7efe840b1910 con 0x7efe8409ac40 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 >> v1:172.21.15.100:6790/0 conn(0x7efe840af1d0 legacy=0x7efe84138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 >> v1:172.21.15.100:6789/0 conn(0x7efe84139370 legacy=0x7efe841d70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 monclient: found mon.b 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe84139970 con 0x7efe8409ac40 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient: _finish_auth 0 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:24.632 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe8c796640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe8c796640 10 monclient: _renew_subs 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe8c796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe8c796640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe841d77f0 con 0x7efe8409ac40 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe8c796640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe8c796640 10 monclient: _renew_subs 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe8c796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe8c796640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe841d7d70 con 0x7efe8409ac40 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe8c796640 1 librados: init done 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe74004bf0 con 0x7efe8409ac40 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7efe74004f00 con 0x7efe8409ac40 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe88d2f640 4 set_mon_vals no callback set 2024-03-29T11:36:24.633 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 monclient: dump: 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efe740053f0 con 0x7efe8409ac40 2024-03-29T11:36:24.634 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3888793360 0 0) 0x7efe74006820 con 0x7efe8409ac40 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 10 client.4117.objecter ms_dispatch 0x7efe84000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 3 client.4117.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.616+0000 7efe7affd640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.617+0000 7efe8c796640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.617+0000 7efe8c796640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.617+0000 7efe8c796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.617+0000 7efe8c796640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6c005080 con 0x7efe8409ac40 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.619+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efe74006a80 con 0x7efe8409ac40 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.619+0000 7efe7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.635 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.619+0000 7efe7affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:24.690 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.691+0000 7efe8c796640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bafc88bc-7772-46af-81dc-9ab1f21e7bbd", "id": 0}] 2024-03-29T11:36:24.690 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.691+0000 7efe8c796640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bafc88bc-7772-46af-81dc-9ab1f21e7bbd", "id": 0}] 2024-03-29T11:36:24.690 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.691+0000 7efe8c796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.690 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.691+0000 7efe8c796640 1 -- 172.21.15.191:0/1018249811 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bafc88bc-7772-46af-81dc-9ab1f21e7bbd", "id": 0} v 0) v1 -- 0x7efe6c0058b0 con 0x7efe8409ac40 2024-03-29T11:36:24.695 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.696+0000 7efe7affd640 1 -- 172.21.15.191:0/1018249811 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bafc88bc-7772-46af-81dc-9ab1f21e7bbd", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3019267833 0 732707515) 0x7efe74005910 con 0x7efe8409ac40 2024-03-29T11:36:24.695 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.696+0000 7efe7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bafc88bc-7772-46af-81dc-9ab1f21e7bbd", "id": 0}] 2024-03-29T11:36:24.695 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.696+0000 7efe7affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:24.695 INFO:teuthology.orchestra.run.smithi191.stdout:0 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 10 librados: watch_flush enter 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 10 librados: watch_flush exit 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 10 monclient: shutdown 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 1 -- 172.21.15.191:0/1018249811 >> v1:172.21.15.191:6789/0 conn(0x7efe8409ac40 legacy=0x7efe840afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 1 -- 172.21.15.191:0/1018249811 shutdown_connections 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 1 -- 172.21.15.191:0/1018249811 >> 172.21.15.191:0/1018249811 conn(0x7efe84066b00 msgr2=0x7efe840b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 1 -- 172.21.15.191:0/1018249811 shutdown_connections 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 1 -- 172.21.15.191:0/1018249811 wait complete. 2024-03-29T11:36:24.697 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.697+0000 7efe8c796640 1 librados: shutdown 2024-03-29T11:36:24.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 0a9f0aba-7c68-4b3a-b9cb-edbb25290719 1 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.807+0000 7f54ba932640 10 monclient: get_monmap_and_config 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.807+0000 7f54ba932640 10 monclient: build_initial_monmap 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.807+0000 7f54ba932640 10 monclient: monmap: 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:36:24.808017+0000 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:36:24.808017+0000 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:24.806 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:24.807 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:24.807 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.807+0000 7f54ba932640 1 Processor -- start 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.807+0000 7f54ba932640 1 -- start start 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.807+0000 7f54ba932640 10 monclient: init 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54ba932640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54ba932640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54ba932640 10 monclient(hunting): picked mon.noname-b con 0x7f54b409ac40 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54ba932640 10 monclient(hunting): picked mon.noname-a con 0x7f54b409c870 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54ba932640 10 monclient(hunting): picked mon.noname-c con 0x7f54b40b4360 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54ba932640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b409dec0 con 0x7f54b409c870 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54ba932640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b409da30 con 0x7f54b40b4360 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54ba932640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b40b39c0 con 0x7f54b409ac40 2024-03-29T11:36:24.816 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54ba932640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54ba932640 10 monclient(hunting): authenticate will time out at 895.125544s 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b8ecf640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f54b40b4360 0x7f54b414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49846/0 (socket says 172.21.15.191:49846) 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b8ecf640 1 -- 172.21.15.191:0/393664212 learned_addr learned my addr 172.21.15.191:0/393664212 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010123561 0 0) 0x7f54b409da30 con 0x7f54b40b4360 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f549c003520 con 0x7f54b40b4360 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232068077 0 0) 0x7f54a4001110 con 0x7f54b409c870 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:36:24.817 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b409da30 con 0x7f54b409c870 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504300119 0 0) 0x7f54b40b39c0 con 0x7f54b409ac40 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a4001110 con 0x7f54b409ac40 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.808+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701850751 0 0) 0x7f549c003520 con 0x7f54b40b4360 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 >> v1:172.21.15.100:6790/0 conn(0x7f54b409ac40 legacy=0x7f54b40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 >> v1:172.21.15.100:6789/0 conn(0x7f54b409c870 legacy=0x7f54b413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:24.818 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b40b5160 con 0x7f54b40b4360 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f54ac001a10 con 0x7f54b40b4360 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: dump: 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.819 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54ac001bf0 con 0x7f54b40b4360 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 4 set_mon_vals no callback set 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 1 -- 172.21.15.191:0/393664212 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f54ac002000 con 0x7f54b40b4360 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:24.820 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54b2ffd640 10 monclient: dump: 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 10 monclient: shutdown 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 1 -- 172.21.15.191:0/393664212 >> v1:172.21.15.191:6789/0 conn(0x7f54b40b4360 legacy=0x7f54b414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.821 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 1 -- 172.21.15.191:0/393664212 shutdown_connections 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 1 -- 172.21.15.191:0/393664212 >> 172.21.15.191:0/393664212 conn(0x7f54b4066b00 msgr2=0x7f54b4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 1 -- 172.21.15.191:0/393664212 shutdown_connections 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.809+0000 7f54ba932640 1 -- 172.21.15.191:0/393664212 wait complete. 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.810+0000 7f54ba932640 10 monclient: build_initial_monmap 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.810+0000 7f54ba932640 10 monclient: monmap: 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:24.822 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.810+0000 7f54ba932640 1 librados: starting msgr at 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.810+0000 7f54ba932640 1 librados: starting objecter 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.810+0000 7f54ba932640 1 Processor -- start 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.810+0000 7f54ba932640 1 -- start start 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.810+0000 7f54ba932640 1 librados: setting wanted keys 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.810+0000 7f54ba932640 1 librados: calling monclient init 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.810+0000 7f54ba932640 10 monclient: init 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54ba932640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54ba932640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54ba932640 10 monclient(hunting): picked mon.noname-c con 0x7f54b409ac40 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54ba932640 10 monclient(hunting): picked mon.noname-a con 0x7f54b41406b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54ba932640 10 monclient(hunting): picked mon.noname-b con 0x7f54b40b07d0 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:24.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54ba932640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b4142220 con 0x7f54b41406b0 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54ba932640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b41399e0 con 0x7f54b40b07d0 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54ba932640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b4143990 con 0x7f54b409ac40 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54ba932640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54ba932640 10 monclient(hunting): authenticate will time out at 895.128518s 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b8ecf640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f54b40b07d0 0x7f54b41392d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49848/0 (socket says 172.21.15.191:49848) 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b8ecf640 1 -- 172.21.15.191:0/1980075873 learned_addr learned my addr 172.21.15.191:0/1980075873 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 client.?.objecter ms_handle_connect 0x7f54b40b07d0 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 client.?.objecter ms_handle_connect 0x7f54b41406b0 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 client.?.objecter ms_handle_connect 0x7f54b409ac40 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227974829 0 0) 0x7f54b41399e0 con 0x7f54b40b07d0 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.824 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a0004eb0 con 0x7f54b40b07d0 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132495980 0 0) 0x7f54b4142220 con 0x7f54b41406b0 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b41399e0 con 0x7f54b41406b0 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532787075 0 0) 0x7f54b4143990 con 0x7f54b409ac40 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:36:24.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b4142220 con 0x7f54b409ac40 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1908091946 0 0) 0x7f54a0004eb0 con 0x7f54b40b07d0 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54b4143990 con 0x7f54b40b07d0 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f54ac001f40 con 0x7f54b40b07d0 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.826 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.811+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3095363154 0 0) 0x7f54b41399e0 con 0x7f54b41406b0 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54a0004eb0 con 0x7f54b41406b0 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f54a4001920 con 0x7f54b41406b0 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.827 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 512240193 0 0) 0x7f54b4142220 con 0x7f54b409ac40 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54b41399e0 con 0x7f54b409ac40 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f54a8001830 con 0x7f54b409ac40 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.828 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2853820343 0 0) 0x7f54b4143990 con 0x7f54b40b07d0 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 >> v1:172.21.15.100:6790/0 conn(0x7f54b409ac40 legacy=0x7f54b413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 >> v1:172.21.15.100:6789/0 conn(0x7f54b41406b0 legacy=0x7f54b4138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 monclient: found mon.b 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:24.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b413a1b0 con 0x7f54b40b07d0 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54ba932640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54ba932640 10 monclient: _renew_subs 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54ba932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54ba932640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b4141330 con 0x7f54b40b07d0 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54ba932640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54ba932640 10 monclient: _renew_subs 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54ba932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54ba932640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b41426b0 con 0x7f54b40b07d0 2024-03-29T11:36:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54ba932640 1 librados: init done 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54ac004e10 con 0x7f54b40b07d0 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f54ac005120 con 0x7f54b40b07d0 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 monclient: dump: 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:24.831 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b27fc640 4 set_mon_vals no callback set 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f54ac005690 con 0x7f54b40b07d0 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 278827036 0 0) 0x7f54ac006ac0 con 0x7f54b40b07d0 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 10 client.4123.objecter ms_dispatch 0x7f54b4000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.812+0000 7f54b0ff9640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.813+0000 7f54ba932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.813+0000 7f54ba932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.813+0000 7f54ba932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.813+0000 7f54ba932640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b409b020 con 0x7f54b40b07d0 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.815+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f54ac0059e0 con 0x7f54b40b07d0 2024-03-29T11:36:24.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.815+0000 7f54b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.815+0000 7f54b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:24.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.887+0000 7f54ba932640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0a9f0aba-7c68-4b3a-b9cb-edbb25290719", "id": 1}] 2024-03-29T11:36:24.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.887+0000 7f54ba932640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0a9f0aba-7c68-4b3a-b9cb-edbb25290719", "id": 1}] 2024-03-29T11:36:24.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.887+0000 7f54ba932640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:24.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.887+0000 7f54ba932640 1 -- 172.21.15.191:0/1980075873 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0a9f0aba-7c68-4b3a-b9cb-edbb25290719", "id": 1} v 0) v1 -- 0x7f54b404f730 con 0x7f54b40b07d0 2024-03-29T11:36:24.894 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.895+0000 7f54b0ff9640 1 -- 172.21.15.191:0/1980075873 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0a9f0aba-7c68-4b3a-b9cb-edbb25290719", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2567324339 0 940483276) 0x7f54ac005d50 con 0x7f54b40b07d0 2024-03-29T11:36:24.894 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.895+0000 7f54b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0a9f0aba-7c68-4b3a-b9cb-edbb25290719", "id": 1}] 2024-03-29T11:36:24.894 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.895+0000 7f54b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:24.894 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-03-29T11:36:24.895 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.896+0000 7f54ba932640 10 librados: watch_flush enter 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.896+0000 7f54ba932640 10 librados: watch_flush exit 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.896+0000 7f54ba932640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.896+0000 7f54ba932640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.896+0000 7f54ba932640 10 monclient: shutdown 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.896+0000 7f54ba932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.896+0000 7f54ba932640 1 -- 172.21.15.191:0/1980075873 >> v1:172.21.15.191:6789/0 conn(0x7f54b40b07d0 legacy=0x7f54b41392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.896+0000 7f54ba932640 1 -- 172.21.15.191:0/1980075873 shutdown_connections 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.896+0000 7f54ba932640 1 -- 172.21.15.191:0/1980075873 >> 172.21.15.191:0/1980075873 conn(0x7f54b4066b00 msgr2=0x7f54b413b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.897+0000 7f54ba932640 1 -- 172.21.15.191:0/1980075873 shutdown_connections 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.897+0000 7f54ba932640 1 -- 172.21.15.191:0/1980075873 wait complete. 2024-03-29T11:36:24.896 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:24.897+0000 7f54ba932640 1 librados: shutdown 2024-03-29T11:36:24.906 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new c2f9c767-038c-4127-b25a-7ed2e956cef1 2 2024-03-29T11:36:25.007 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.008+0000 7f1108b9f640 10 monclient: get_monmap_and_config 2024-03-29T11:36:25.007 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.008+0000 7f1108b9f640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.007 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.008+0000 7f1108b9f640 10 monclient: monmap: 2024-03-29T11:36:25.007 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:25.007 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.008 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:36:25.009516+0000 2024-03-29T11:36:25.008 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:36:25.009516+0000 2024-03-29T11:36:25.008 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.008 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.008 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:25.008 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:25.008 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:25.008 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.015 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 1 Processor -- start 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 1 -- start start 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 10 monclient: init 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 10 monclient(hunting): picked mon.noname-c con 0x7f1104145610 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 10 monclient(hunting): picked mon.noname-a con 0x7f1104146050 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 10 monclient(hunting): picked mon.noname-b con 0x7f110413c030 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040aacc0 con 0x7f1104146050 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040b2720 con 0x7f1104145610 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040aaf30 con 0x7f110413c030 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.009+0000 7f1108b9f640 10 monclient(hunting): authenticate will time out at 895.327003s 2024-03-29T11:36:25.016 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11037fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f1104145610 0x7f11040b2c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49854/0 (socket says 172.21.15.191:49854) 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11037fe640 1 -- 172.21.15.191:0/3965507005 learned_addr learned my addr 172.21.15.191:0/3965507005 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215345210 0 0) 0x7f11040b2720 con 0x7f1104145610 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ec003520 con 0x7f1104145610 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629297677 0 0) 0x7f11040aacc0 con 0x7f1104146050 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11040b2720 con 0x7f1104146050 2024-03-29T11:36:25.017 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606328947 0 0) 0x7f11040aaf30 con 0x7f110413c030 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11040aacc0 con 0x7f110413c030 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1047357728 0 0) 0x7f10ec003520 con 0x7f1104145610 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 >> v1:172.21.15.100:6790/0 conn(0x7f110413c030 legacy=0x7f110414d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 >> v1:172.21.15.100:6789/0 conn(0x7f1104146050 legacy=0x7f110413b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 monclient: found mon.noname-c 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11040a9b20 con 0x7f1104145610 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f10f4001690 con 0x7f1104145610 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f1108b9f640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f1108b9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: dump: 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.019 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f1108b9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10f4001870 con 0x7f1104145610 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f1108b9f640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 1 -- 172.21.15.191:0/3965507005 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f10f4001c00 con 0x7f1104145610 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f1108b9f640 4 set_mon_vals no callback set 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:25.020 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.010+0000 7f11027fc640 10 monclient: dump: 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.011+0000 7f1108b9f640 10 monclient: shutdown 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.011+0000 7f1108b9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.011+0000 7f1108b9f640 1 -- 172.21.15.191:0/3965507005 >> v1:172.21.15.191:6789/0 conn(0x7f1104145610 legacy=0x7f11040b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.021 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.011+0000 7f1108b9f640 1 -- 172.21.15.191:0/3965507005 shutdown_connections 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.011+0000 7f1108b9f640 1 -- 172.21.15.191:0/3965507005 >> 172.21.15.191:0/3965507005 conn(0x7f11040726e0 msgr2=0x7f1104066920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.011+0000 7f1108b9f640 1 -- 172.21.15.191:0/3965507005 shutdown_connections 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.011+0000 7f1108b9f640 1 -- 172.21.15.191:0/3965507005 wait complete. 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.011+0000 7f1108b9f640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.011+0000 7f1108b9f640 10 monclient: monmap: 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:25.022 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.012+0000 7f1108b9f640 1 librados: starting msgr at 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.012+0000 7f1108b9f640 1 librados: starting objecter 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.012+0000 7f1108b9f640 1 Processor -- start 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.012+0000 7f1108b9f640 1 -- start start 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.012+0000 7f1108b9f640 1 librados: setting wanted keys 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.012+0000 7f1108b9f640 1 librados: calling monclient init 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.012+0000 7f1108b9f640 10 monclient: init 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.012+0000 7f1108b9f640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.012+0000 7f1108b9f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.012+0000 7f1108b9f640 10 monclient(hunting): picked mon.noname-a con 0x7f110413c030 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 10 monclient(hunting): picked mon.noname-c con 0x7f1104145610 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 10 monclient(hunting): picked mon.noname-b con 0x7f1104146050 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040acfd0 con 0x7f110413c030 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11041d70b0 con 0x7f1104146050 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11040ad240 con 0x7f1104145610 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 10 monclient(hunting): authenticate will time out at 895.330153s 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1103fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f1104146050 0x7f11040b0a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49866/0 (socket says 172.21.15.191:49866) 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1103fff640 1 -- 172.21.15.191:0/337723919 learned_addr learned my addr 172.21.15.191:0/337723919 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f11037fe640 1 --1- 172.21.15.191:0/337723919 >> v1:172.21.15.100:6789/0 conn(0x7f110413c030 0x7f1104142700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.191:58662/0 (socket says 172.21.15.191:58662) 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 client.?.objecter ms_handle_connect 0x7f1104146050 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 client.?.objecter ms_handle_connect 0x7f110413c030 2024-03-29T11:36:25.024 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 client.?.objecter ms_handle_connect 0x7f1104145610 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704571683 0 0) 0x7f11041d70b0 con 0x7f1104146050 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f0004eb0 con 0x7f1104146050 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192206585 0 0) 0x7f11040ad240 con 0x7f1104145610 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:36:25.025 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11041d70b0 con 0x7f1104145610 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294557279 0 0) 0x7f11040acfd0 con 0x7f110413c030 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11040ad240 con 0x7f110413c030 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2171854305 0 0) 0x7f11041d70b0 con 0x7f1104145610 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11040acfd0 con 0x7f1104145610 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3512570325 0 0) 0x7f10f0004eb0 con 0x7f1104146050 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11041d70b0 con 0x7f1104146050 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f10f8001710 con 0x7f1104145610 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): dump: 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f10fc001c50 con 0x7f1104146050 2024-03-29T11:36:25.027 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): dump: 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3997396108 0 0) 0x7f11040ad240 con 0x7f110413c030 2024-03-29T11:36:25.028 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10f0004eb0 con 0x7f110413c030 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f10f40019e0 con 0x7f110413c030 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient(hunting): dump: 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.029 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2747007889 0 0) 0x7f11040acfd0 con 0x7f1104145610 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 >> v1:172.21.15.191:6789/0 conn(0x7f1104146050 legacy=0x7f11040b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 >> v1:172.21.15.100:6789/0 conn(0x7f110413c030 legacy=0x7f1104142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 monclient: found mon.c 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110413fc20 con 0x7f1104145610 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 5 monclient: authenticate success, global_id 4121 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 10 monclient: _renew_subs 2024-03-29T11:36:25.030 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11041d7ca0 con 0x7f1104145610 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 10 monclient: _renew_subs 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11040abcf0 con 0x7f1104145610 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f1108b9f640 1 librados: init done 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.013+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10f8001890 con 0x7f1104145610 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f10f8001d80 con 0x7f1104145610 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f1101ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 10 monclient: dump: 2024-03-29T11:36:25.031 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f10f8003ba0 con 0x7f1104145610 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1036475448 0 0) 0x7f10f80052a0 con 0x7f1104145610 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 10 client.4121.objecter ms_dispatch 0x7f1104000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 3 client.4121.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 3 client.4121.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:36:25.032 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.014+0000 7f10ebfff640 20 client.4121.objecter dump_active .. 0 homeless 2024-03-29T11:36:25.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.015+0000 7f1108b9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.015+0000 7f1108b9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.015+0000 7f1108b9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:25.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.015+0000 7f1108b9f640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e4005080 con 0x7f1104145610 2024-03-29T11:36:25.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.017+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f10f8004f90 con 0x7f1104145610 2024-03-29T11:36:25.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.017+0000 7f10ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.033 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.017+0000 7f10ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:25.049 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:25.050+0000 7fda32e32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:36:25.087 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.088+0000 7f1108b9f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c2f9c767-038c-4127-b25a-7ed2e956cef1", "id": 2}] 2024-03-29T11:36:25.087 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.088+0000 7f1108b9f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c2f9c767-038c-4127-b25a-7ed2e956cef1", "id": 2}] 2024-03-29T11:36:25.087 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.088+0000 7f1108b9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:25.087 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.088+0000 7f1108b9f640 1 -- 172.21.15.191:0/337723919 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd new", "uuid": "c2f9c767-038c-4127-b25a-7ed2e956cef1", "id": 2} v 0) v1 -- 0x7f10e40058b0 con 0x7f1104145610 2024-03-29T11:36:25.089 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:25.090+0000 7f1b76fdb040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:36:25.089 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.091+0000 7f10ebfff640 1 -- 172.21.15.191:0/337723919 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c2f9c767-038c-4127-b25a-7ed2e956cef1", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3108932444 0 216599125) 0x7f10f8004150 con 0x7f1104145610 2024-03-29T11:36:25.089 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.091+0000 7f10ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c2f9c767-038c-4127-b25a-7ed2e956cef1", "id": 2}] 2024-03-29T11:36:25.089 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.091+0000 7f10ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:25.090 INFO:teuthology.orchestra.run.smithi191.stdout:2 2024-03-29T11:36:25.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.091+0000 7f1108b9f640 10 librados: watch_flush enter 2024-03-29T11:36:25.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.091+0000 7f1108b9f640 10 librados: watch_flush exit 2024-03-29T11:36:25.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.091+0000 7f1108b9f640 20 client.4121.objecter shutdown clearing up homeless session... 2024-03-29T11:36:25.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.091+0000 7f1108b9f640 10 client.4121.objecter successfully canceled tick 2024-03-29T11:36:25.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.092+0000 7f1108b9f640 10 monclient: shutdown 2024-03-29T11:36:25.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.092+0000 7f1108b9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.092+0000 7f1108b9f640 1 -- 172.21.15.191:0/337723919 >> v1:172.21.15.100:6790/0 conn(0x7f1104145610 legacy=0x7f11041d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.092+0000 7f1108b9f640 1 -- 172.21.15.191:0/337723919 shutdown_connections 2024-03-29T11:36:25.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.092+0000 7f1108b9f640 1 -- 172.21.15.191:0/337723919 >> 172.21.15.191:0/337723919 conn(0x7f11040726e0 msgr2=0x7f1104143540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.091 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.092+0000 7f1108b9f640 1 -- 172.21.15.191:0/337723919 shutdown_connections 2024-03-29T11:36:25.092 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.092+0000 7f1108b9f640 1 -- 172.21.15.191:0/337723919 wait complete. 2024-03-29T11:36:25.092 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.092+0000 7f1108b9f640 1 librados: shutdown 2024-03-29T11:36:25.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 75820eae-51d4-4b75-b4be-f45edc929288 3 2024-03-29T11:36:25.104 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:25.106+0000 7fda32e32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:36:25.144 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:25.145+0000 7f1b76fdb040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:36:25.152 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:25.153+0000 7fda32e32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:36:25.188 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:25.189+0000 7f1b76fdb040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.202+0000 7ff8bf26e640 10 monclient: get_monmap_and_config 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.202+0000 7ff8bf26e640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.202+0000 7ff8bf26e640 10 monclient: monmap: 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:36:25.203238+0000 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:36:25.203238+0000 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:25.201 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:25.202 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:25.202 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.202+0000 7ff8bf26e640 1 Processor -- start 2024-03-29T11:36:25.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 1 -- start start 2024-03-29T11:36:25.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 10 monclient: init 2024-03-29T11:36:25.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:25.212 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 10 monclient(hunting): picked mon.noname-c con 0x7ff8b80af360 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 10 monclient(hunting): picked mon.noname-b con 0x7ff8b806af60 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 10 monclient(hunting): picked mon.noname-a con 0x7ff8b8145e60 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b80b0520 con 0x7ff8b8145e60 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b80aff20 con 0x7ff8b80af360 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b80b1da0 con 0x7ff8b806af60 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bf26e640 10 monclient(hunting): authenticate will time out at 895.520882s 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bd00a640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ff8b80af360 0x7ff8b806c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49876/0 (socket says 172.21.15.191:49876) 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.203+0000 7ff8bd00a640 1 -- 172.21.15.191:0/1133844216 learned_addr learned my addr 172.21.15.191:0/1133844216 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821569213 0 0) 0x7ff8b80aff20 con 0x7ff8b80af360 2024-03-29T11:36:25.213 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a4003520 con 0x7ff8b80af360 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294073071 0 0) 0x7ff8b80b1da0 con 0x7ff8b806af60 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b80aff20 con 0x7ff8b806af60 2024-03-29T11:36:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610495633 0 0) 0x7ff8b80b0520 con 0x7ff8b8145e60 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b80b1da0 con 0x7ff8b8145e60 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1581052011 0 0) 0x7ff8a4003520 con 0x7ff8b80af360 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 >> v1:172.21.15.100:6790/0 conn(0x7ff8b806af60 legacy=0x7ff8b8145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 >> v1:172.21.15.100:6789/0 conn(0x7ff8b8145e60 legacy=0x7ff8b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 monclient: found mon.noname-c 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b80b2080 con 0x7ff8b80af360 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.215 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8bf26e640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8bf26e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff8a8001690 con 0x7ff8b80af360 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: dump: 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.216 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.217 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.217 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.217 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8bf26e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8a8001870 con 0x7ff8b80af360 2024-03-29T11:36:25.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 1 -- 172.21.15.191:0/1133844216 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff8a8001c00 con 0x7ff8b80af360 2024-03-29T11:36:25.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:25.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8affff640 10 monclient: dump: 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8bf26e640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.204+0000 7ff8bf26e640 4 set_mon_vals no callback set 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.205+0000 7ff8bf26e640 10 monclient: shutdown 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.205+0000 7ff8bf26e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.205+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1133844216 >> v1:172.21.15.191:6789/0 conn(0x7ff8b80af360 legacy=0x7ff8b806c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.205+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1133844216 shutdown_connections 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.205+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1133844216 >> 172.21.15.191:0/1133844216 conn(0x7ff8b80b58f0 msgr2=0x7ff8b80b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.205+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1133844216 shutdown_connections 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.205+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1133844216 wait complete. 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.206+0000 7ff8bf26e640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.206+0000 7ff8bf26e640 10 monclient: monmap: 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:25.219 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.206+0000 7ff8bf26e640 1 librados: starting msgr at 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.206+0000 7ff8bf26e640 1 librados: starting objecter 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.206+0000 7ff8bf26e640 1 Processor -- start 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.206+0000 7ff8bf26e640 1 -- start start 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.206+0000 7ff8bf26e640 1 librados: setting wanted keys 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.206+0000 7ff8bf26e640 1 librados: calling monclient init 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.206+0000 7ff8bf26e640 10 monclient: init 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bf26e640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bf26e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bf26e640 10 monclient(hunting): picked mon.noname-a con 0x7ff8b806af60 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bf26e640 10 monclient(hunting): picked mon.noname-c con 0x7ff8b80af360 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:25.220 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bf26e640 10 monclient(hunting): picked mon.noname-b con 0x7ff8b8145e60 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bf26e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b813d200 con 0x7ff8b806af60 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bf26e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b811e0c0 con 0x7ff8b8145e60 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bf26e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b813c3c0 con 0x7ff8b80af360 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bf26e640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bf26e640 10 monclient(hunting): authenticate will time out at 895.524736s 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bd80b640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ff8b8145e60 0x7ff8b81da790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49890/0 (socket says 172.21.15.191:49890) 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8bd80b640 1 -- 172.21.15.191:0/1371048490 learned_addr learned my addr 172.21.15.191:0/1371048490 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 client.?.objecter ms_handle_connect 0x7ff8b8145e60 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 client.?.objecter ms_handle_connect 0x7ff8b806af60 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 client.?.objecter ms_handle_connect 0x7ff8b80af360 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.221 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099039739 0 0) 0x7ff8b811e0c0 con 0x7ff8b8145e60 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89c004eb0 con 0x7ff8b8145e60 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863961809 0 0) 0x7ff8b813d200 con 0x7ff8b806af60 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.207+0000 7ff8adffb640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b811e0c0 con 0x7ff8b806af60 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743893656 0 0) 0x7ff8b813c3c0 con 0x7ff8b80af360 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.222 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b813d200 con 0x7ff8b80af360 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3862328976 0 0) 0x7ff89c004eb0 con 0x7ff8b8145e60 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8b813c3c0 con 0x7ff8b8145e60 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff8b0001be0 con 0x7ff8b8145e60 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): dump: 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.223 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 167936377 0 0) 0x7ff8b811e0c0 con 0x7ff8b806af60 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff89c004eb0 con 0x7ff8b806af60 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3468726043 0 0) 0x7ff8b813d200 con 0x7ff8b80af360 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8b811e0c0 con 0x7ff8b80af360 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff8a8001b80 con 0x7ff8b806af60 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): dump: 2024-03-29T11:36:25.224 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff8b40018b0 con 0x7ff8b80af360 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient(hunting): dump: 2024-03-29T11:36:25.225 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 788779325 0 0) 0x7ff8b813c3c0 con 0x7ff8b8145e60 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 >> v1:172.21.15.100:6790/0 conn(0x7ff8b80af360 legacy=0x7ff8b813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 >> v1:172.21.15.100:6789/0 conn(0x7ff8b806af60 legacy=0x7ff8b813bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.226 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 monclient: found mon.b 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b813c550 con 0x7ff8b8145e60 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8bf26e640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8bf26e640 10 monclient: _renew_subs 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8bf26e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b80717b0 con 0x7ff8b8145e60 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8b0003990 con 0x7ff8b8145e60 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff8b0003d60 con 0x7ff8b8145e60 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 monclient: dump: 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff8b00042d0 con 0x7ff8b8145e60 2024-03-29T11:36:25.228 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8af7fe640 4 set_mon_vals no callback set 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8bf26e640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8bf26e640 10 monclient: _renew_subs 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8bf26e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b813b190 con 0x7ff8b8145e60 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8bf26e640 1 librados: init done 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 407756383 0 0) 0x7ff8b0006630 con 0x7ff8b8145e60 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 10 client.4135.objecter ms_dispatch 0x7ff8b8000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.208+0000 7ff8adffb640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.209+0000 7ff8bf26e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.209+0000 7ff8bf26e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.209+0000 7ff8bf26e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.209+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b8067c00 con 0x7ff8b8145e60 2024-03-29T11:36:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.211+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff8b0005b40 con 0x7ff8b8145e60 2024-03-29T11:36:25.230 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.211+0000 7ff8adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.230 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.211+0000 7ff8adffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:25.230 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:25.211+0000 7fda32e32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:36:25.241 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:25.243+0000 7f1b76fdb040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:36:25.254 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:25.255+0000 7fda32e32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:36:25.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.284+0000 7ff8bf26e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "75820eae-51d4-4b75-b4be-f45edc929288", "id": 3}] 2024-03-29T11:36:25.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.284+0000 7ff8bf26e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "75820eae-51d4-4b75-b4be-f45edc929288", "id": 3}] 2024-03-29T11:36:25.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.284+0000 7ff8bf26e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.283 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.284+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1371048490 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd new", "uuid": "75820eae-51d4-4b75-b4be-f45edc929288", "id": 3} v 0) v1 -- 0x7ff8b804f730 con 0x7ff8b8145e60 2024-03-29T11:36:25.284 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:25.285+0000 7f1b76fdb040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:36:25.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.292+0000 7ff8adffb640 1 -- 172.21.15.191:0/1371048490 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "75820eae-51d4-4b75-b4be-f45edc929288", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 596912325 0 525046306) 0x7ff8b0005d60 con 0x7ff8b8145e60 2024-03-29T11:36:25.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.292+0000 7ff8adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "75820eae-51d4-4b75-b4be-f45edc929288", "id": 3}] 2024-03-29T11:36:25.291 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.292+0000 7ff8adffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:25.291 INFO:teuthology.orchestra.run.smithi191.stdout:3 2024-03-29T11:36:25.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.292+0000 7ff8bf26e640 10 librados: watch_flush enter 2024-03-29T11:36:25.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.292+0000 7ff8bf26e640 10 librados: watch_flush exit 2024-03-29T11:36:25.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.292+0000 7ff8bf26e640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T11:36:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.292+0000 7ff8bf26e640 10 client.4135.objecter successfully canceled tick 2024-03-29T11:36:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.293+0000 7ff8bf26e640 10 monclient: shutdown 2024-03-29T11:36:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.293+0000 7ff8bf26e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.293+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1371048490 >> v1:172.21.15.191:6789/0 conn(0x7ff8b8145e60 legacy=0x7ff8b81da790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.293+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1371048490 shutdown_connections 2024-03-29T11:36:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.293+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1371048490 >> 172.21.15.191:0/1371048490 conn(0x7ff8b80b58f0 msgr2=0x7ff8b80b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.293+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1371048490 shutdown_connections 2024-03-29T11:36:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.293+0000 7ff8bf26e640 1 -- 172.21.15.191:0/1371048490 wait complete. 2024-03-29T11:36:25.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.293+0000 7ff8bf26e640 1 librados: shutdown 2024-03-29T11:36:25.302 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 655e63aa-3727-409b-b9e2-652246de4657 4 2024-03-29T11:36:25.375 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:25.377+0000 7fda32e32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:36:25.399 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:25.400+0000 7f1b76fdb040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:36:25.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.443+0000 7f837b9ba640 10 monclient: get_monmap_and_config 2024-03-29T11:36:25.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.443+0000 7f837b9ba640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.442 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.443+0000 7f837b9ba640 10 monclient: monmap: 2024-03-29T11:36:25.442 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:25.442 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.442 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:36:25.444097+0000 2024-03-29T11:36:25.442 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:36:25.444097+0000 2024-03-29T11:36:25.442 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.443 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.443 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:25.443 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:25.443 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:25.443 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.443+0000 7f837b9ba640 1 Processor -- start 2024-03-29T11:36:25.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.443+0000 7f837b9ba640 1 -- start start 2024-03-29T11:36:25.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.443+0000 7f837b9ba640 10 monclient: init 2024-03-29T11:36:25.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f837b9ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f837b9ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f837b9ba640 10 monclient(hunting): picked mon.noname-c con 0x7f837409ac40 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f837b9ba640 10 monclient(hunting): picked mon.noname-b con 0x7f837409c870 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f837b9ba640 10 monclient(hunting): picked mon.noname-a con 0x7f83740b4360 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f837b9ba640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837409dec0 con 0x7f83740b4360 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f837b9ba640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837409da30 con 0x7f837409ac40 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f837b9ba640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83740b39c0 con 0x7f837409c870 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f837b9ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f837b9ba640 10 monclient(hunting): authenticate will time out at 895.761636s 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f8379756640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f837409ac40 0x7f83740b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49898/0 (socket says 172.21.15.191:49898) 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f8379756640 1 -- 172.21.15.191:0/3791402928 learned_addr learned my addr 172.21.15.191:0/3791402928 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939396579 0 0) 0x7f837409da30 con 0x7f837409ac40 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f836bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f836bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f836bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f836bfff640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.444+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8360003520 con 0x7f837409ac40 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528332023 0 0) 0x7f836c001490 con 0x7f83740b4360 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f837409da30 con 0x7f83740b4360 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607693835 0 0) 0x7f83740b39c0 con 0x7f837409c870 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:36:25.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f836c001490 con 0x7f837409c870 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3946361824 0 0) 0x7f8360003520 con 0x7f837409ac40 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 >> v1:172.21.15.100:6790/0 conn(0x7f837409c870 legacy=0x7f837413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 >> v1:172.21.15.100:6789/0 conn(0x7f83740b4360 legacy=0x7f837414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 monclient: found mon.noname-c 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83740b5160 con 0x7f837409ac40 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8364001690 con 0x7f837409ac40 2024-03-29T11:36:25.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f837b9ba640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f837b9ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: dump: 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.455 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f837b9ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8364001870 con 0x7f837409ac40 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 1 -- 172.21.15.191:0/3791402928 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8364001c80 con 0x7f837409ac40 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f837b9ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f837b9ba640 4 set_mon_vals no callback set 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.445+0000 7f836bfff640 10 monclient: dump: 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.456 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.446+0000 7f837b9ba640 10 monclient: shutdown 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.446+0000 7f837b9ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.446+0000 7f837b9ba640 1 -- 172.21.15.191:0/3791402928 >> v1:172.21.15.191:6789/0 conn(0x7f837409ac40 legacy=0x7f83740b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.446+0000 7f837b9ba640 1 -- 172.21.15.191:0/3791402928 shutdown_connections 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.446+0000 7f837b9ba640 1 -- 172.21.15.191:0/3791402928 >> 172.21.15.191:0/3791402928 conn(0x7f8374066b00 msgr2=0x7f8374066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.446+0000 7f837b9ba640 1 -- 172.21.15.191:0/3791402928 shutdown_connections 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.446+0000 7f837b9ba640 1 -- 172.21.15.191:0/3791402928 wait complete. 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.447+0000 7f837b9ba640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.447+0000 7f837b9ba640 10 monclient: monmap: 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.447+0000 7f837b9ba640 1 librados: starting msgr at 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.447+0000 7f837b9ba640 1 librados: starting objecter 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.447+0000 7f837b9ba640 1 Processor -- start 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.447+0000 7f837b9ba640 1 -- start start 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.447+0000 7f837b9ba640 1 librados: setting wanted keys 2024-03-29T11:36:25.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.447+0000 7f837b9ba640 1 librados: calling monclient init 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.447+0000 7f837b9ba640 10 monclient: init 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f837b9ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f837b9ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f837b9ba640 10 monclient(hunting): picked mon.noname-b con 0x7f837409ac40 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f837b9ba640 10 monclient(hunting): picked mon.noname-c con 0x7f83740b0210 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f837b9ba640 10 monclient(hunting): picked mon.noname-a con 0x7f83741d7c90 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f837b9ba640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83740afb50 con 0x7f83741d7c90 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f837b9ba640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83741d70f0 con 0x7f837409ac40 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f837b9ba640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83740b0650 con 0x7f83740b0210 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f837b9ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f837b9ba640 10 monclient(hunting): authenticate will time out at 895.765841s 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8379756640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f837409ac40 0x7f83740af440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49912/0 (socket says 172.21.15.191:49912) 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8379756640 1 -- 172.21.15.191:0/1392755958 learned_addr learned my addr 172.21.15.191:0/1392755958 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:25.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 10 client.?.objecter ms_handle_connect 0x7f837409ac40 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658929998 0 0) 0x7f83741d70f0 con 0x7f837409ac40 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8358004eb0 con 0x7f837409ac40 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 10 client.?.objecter ms_handle_connect 0x7f83741d7c90 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 10 client.?.objecter ms_handle_connect 0x7f83740b0210 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.448+0000 7f8369ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007651551 0 0) 0x7f83740afb50 con 0x7f83741d7c90 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83741d70f0 con 0x7f83741d7c90 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370102466 0 0) 0x7f83740b0650 con 0x7f83740b0210 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83740afb50 con 0x7f83740b0210 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2302147475 0 0) 0x7f8358004eb0 con 0x7f837409ac40 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83740b0650 con 0x7f837409ac40 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8364001bc0 con 0x7f837409ac40 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:25.461 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3858243092 0 0) 0x7f83741d70f0 con 0x7f83741d7c90 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8358004eb0 con 0x7f83741d7c90 2024-03-29T11:36:25.462 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f836c001a80 con 0x7f83741d7c90 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2617969579 0 0) 0x7f83740afb50 con 0x7f83740b0210 2024-03-29T11:36:25.463 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83741d70f0 con 0x7f83740b0210 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8370001870 con 0x7f83740b0210 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.464 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1709355954 0 0) 0x7f83740b0650 con 0x7f837409ac40 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 >> v1:172.21.15.100:6790/0 conn(0x7f83740b0210 legacy=0x7f83741d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 >> v1:172.21.15.100:6789/0 conn(0x7f83741d7c90 legacy=0x7f83741e1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 monclient: found mon.b 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83740acd00 con 0x7f837409ac40 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f837b9ba640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f837b9ba640 10 monclient: _renew_subs 2024-03-29T11:36:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f837b9ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f837b9ba640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83740b18a0 con 0x7f837409ac40 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8364004bf0 con 0x7f837409ac40 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8364004f00 con 0x7f837409ac40 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 monclient: dump: 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.466 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8364005470 con 0x7f837409ac40 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f836b7fe640 4 set_mon_vals no callback set 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f837b9ba640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f837b9ba640 10 monclient: _renew_subs 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f837b9ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f837b9ba640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83741e3570 con 0x7f837409ac40 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f837b9ba640 1 librados: init done 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1269739553 0 0) 0x7f8364005eb0 con 0x7f837409ac40 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 10 client.4141.objecter ms_dispatch 0x7f8374000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:36:25.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.449+0000 7f8369ffb640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:36:25.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.450+0000 7f837b9ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.450+0000 7f837b9ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.450+0000 7f837b9ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.450+0000 7f837b9ba640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837409b020 con 0x7f837409ac40 2024-03-29T11:36:25.468 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:25.452+0000 7fda32e32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:36:25.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.452+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8364006070 con 0x7f837409ac40 2024-03-29T11:36:25.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.452+0000 7f8369ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.468 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.452+0000 7f8369ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:25.469 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:25.470+0000 7f1b76fdb040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:36:25.497 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:25.498+0000 7fda32e32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:36:25.515 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:25.517+0000 7f1b76fdb040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:36:25.521 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.522+0000 7f837b9ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "655e63aa-3727-409b-b9e2-652246de4657", "id": 4}] 2024-03-29T11:36:25.521 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.522+0000 7f837b9ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "655e63aa-3727-409b-b9e2-652246de4657", "id": 4}] 2024-03-29T11:36:25.521 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.522+0000 7f837b9ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.521 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.522+0000 7f837b9ba640 1 -- 172.21.15.191:0/1392755958 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd new", "uuid": "655e63aa-3727-409b-b9e2-652246de4657", "id": 4} v 0) v1 -- 0x7f837413e190 con 0x7f837409ac40 2024-03-29T11:36:25.526 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.528+0000 7f8369ffb640 1 -- 172.21.15.191:0/1392755958 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "655e63aa-3727-409b-b9e2-652246de4657", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 572121385 0 1697012583) 0x7f83640059a0 con 0x7f837409ac40 2024-03-29T11:36:25.527 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.528+0000 7f8369ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "655e63aa-3727-409b-b9e2-652246de4657", "id": 4}] 2024-03-29T11:36:25.527 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.528+0000 7f8369ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:25.527 INFO:teuthology.orchestra.run.smithi191.stdout:4 2024-03-29T11:36:25.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.528+0000 7f837b9ba640 10 librados: watch_flush enter 2024-03-29T11:36:25.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 10 librados: watch_flush exit 2024-03-29T11:36:25.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:36:25.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:36:25.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 10 monclient: shutdown 2024-03-29T11:36:25.528 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 1 -- 172.21.15.191:0/1392755958 >> v1:172.21.15.191:6789/0 conn(0x7f837409ac40 legacy=0x7f83740af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 1 -- 172.21.15.191:0/1392755958 shutdown_connections 2024-03-29T11:36:25.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 1 -- 172.21.15.191:0/1392755958 >> 172.21.15.191:0/1392755958 conn(0x7f8374066b00 msgr2=0x7f83740b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 1 -- 172.21.15.191:0/1392755958 shutdown_connections 2024-03-29T11:36:25.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 1 -- 172.21.15.191:0/1392755958 wait complete. 2024-03-29T11:36:25.529 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.529+0000 7f837b9ba640 1 librados: shutdown 2024-03-29T11:36:25.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new fee1d4d7-1159-4388-ad5f-82e515d74b23 5 2024-03-29T11:36:25.551 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:25.552+0000 7fda32e32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:36:25.569 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:25.570+0000 7f1b76fdb040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:36:25.601 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:25.602+0000 7fda32e32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:36:25.616 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:25.617+0000 7f1b76fdb040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:36:25.650 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.652+0000 7f2bee6ba640 10 monclient: get_monmap_and_config 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.652+0000 7f2bee6ba640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.652+0000 7f2bee6ba640 10 monclient: monmap: 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:36:25.653062+0000 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:36:25.653062+0000 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:25.651 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:25.652 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.660 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.652+0000 7f2bee6ba640 1 Processor -- start 2024-03-29T11:36:25.660 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.652+0000 7f2bee6ba640 1 -- start start 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.652+0000 7f2bee6ba640 10 monclient: init 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2bee6ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2bee6ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2bee6ba640 10 monclient(hunting): picked mon.noname-a con 0x7f2be809ac40 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2bee6ba640 10 monclient(hunting): picked mon.noname-c con 0x7f2be809c870 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2bee6ba640 10 monclient(hunting): picked mon.noname-b con 0x7f2be80b4360 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2bee6ba640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be809dec0 con 0x7f2be809ac40 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2bee6ba640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be809da30 con 0x7f2be809c870 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2bee6ba640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be80b39c0 con 0x7f2be80b4360 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2bee6ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2bee6ba640 10 monclient(hunting): authenticate will time out at 895.970660s 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2be77fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f2be809c870 0x7f2be813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49920/0 (socket says 172.21.15.191:49920) 2024-03-29T11:36:25.661 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2be77fe640 1 -- 172.21.15.191:0/3936226474 learned_addr learned my addr 172.21.15.191:0/3936226474 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104440308 0 0) 0x7f2be809da30 con 0x7f2be809c870 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2be6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2be6ffd640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd0003520 con 0x7f2be809c870 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.653+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490205453 0 0) 0x7f2be80b39c0 con 0x7f2be80b4360 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be809da30 con 0x7f2be80b4360 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787207820 0 0) 0x7f2bd8001110 con 0x7f2be809ac40 2024-03-29T11:36:25.662 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be80b39c0 con 0x7f2be809ac40 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3215636676 0 0) 0x7f2bd0003520 con 0x7f2be809c870 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 >> v1:172.21.15.100:6790/0 conn(0x7f2be80b4360 legacy=0x7f2be814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 >> v1:172.21.15.100:6789/0 conn(0x7f2be809ac40 legacy=0x7f2be80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be80b5160 con 0x7f2be809c870 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2bee6ba640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2bdc001690 con 0x7f2be809c870 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2bee6ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: dump: 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.664 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2bee6ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bdc001870 con 0x7f2be809c870 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 1 -- 172.21.15.191:0/3936226474 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2bdc001c80 con 0x7f2be809c870 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2be6ffd640 10 monclient: dump: 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2bee6ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.654+0000 7f2bee6ba640 4 set_mon_vals no callback set 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.655+0000 7f2bee6ba640 10 monclient: shutdown 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.655+0000 7f2bee6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.655+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3936226474 >> v1:172.21.15.191:6789/0 conn(0x7f2be809c870 legacy=0x7f2be813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.655+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3936226474 shutdown_connections 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.655+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3936226474 >> 172.21.15.191:0/3936226474 conn(0x7f2be8066b00 msgr2=0x7f2be8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.655+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3936226474 shutdown_connections 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.655+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3936226474 wait complete. 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.656+0000 7f2bee6ba640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.656+0000 7f2bee6ba640 10 monclient: monmap: 2024-03-29T11:36:25.666 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.656+0000 7f2bee6ba640 1 librados: starting msgr at 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.656+0000 7f2bee6ba640 1 librados: starting objecter 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.656+0000 7f2bee6ba640 1 Processor -- start 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.656+0000 7f2bee6ba640 1 -- start start 2024-03-29T11:36:25.667 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.656+0000 7f2bee6ba640 1 librados: setting wanted keys 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.656+0000 7f2bee6ba640 1 librados: calling monclient init 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.656+0000 7f2bee6ba640 10 monclient: init 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2bee6ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2bee6ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2bee6ba640 10 monclient(hunting): picked mon.noname-c con 0x7f2be809ac40 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2bee6ba640 10 monclient(hunting): picked mon.noname-a con 0x7f2be80b07a0 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2bee6ba640 10 monclient(hunting): picked mon.noname-b con 0x7f2be81d7cd0 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2bee6ba640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be80acd70 con 0x7f2be80b07a0 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2bee6ba640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be81d8100 con 0x7f2be81d7cd0 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2bee6ba640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be811f8d0 con 0x7f2be809ac40 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2bee6ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2bee6ba640 10 monclient(hunting): authenticate will time out at 895.974986s 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2becc57640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f2be81d7cd0 0x7f2be81e1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49922/0 (socket says 172.21.15.191:49922) 2024-03-29T11:36:25.668 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.657+0000 7f2becc57640 1 -- 172.21.15.191:0/3930222204 learned_addr learned my addr 172.21.15.191:0/3930222204 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 client.?.objecter ms_handle_connect 0x7f2be81d7cd0 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 client.?.objecter ms_handle_connect 0x7f2be809ac40 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 client.?.objecter ms_handle_connect 0x7f2be80b07a0 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464138474 0 0) 0x7f2be81d8100 con 0x7f2be81d7cd0 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd4004eb0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726010932 0 0) 0x7f2be80acd70 con 0x7f2be80b07a0 2024-03-29T11:36:25.669 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be81d8100 con 0x7f2be80b07a0 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941793561 0 0) 0x7f2be811f8d0 con 0x7f2be809ac40 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be80acd70 con 0x7f2be809ac40 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2094563803 0 0) 0x7f2bd4004eb0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2be811f8d0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.670 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2be0001c00 con 0x7f2be81d7cd0 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:25.671 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2716130178 0 0) 0x7f2be81d8100 con 0x7f2be80b07a0 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bd4004eb0 con 0x7f2be80b07a0 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2bdc001b80 con 0x7f2be80b07a0 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.672 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3793720274 0 0) 0x7f2be80acd70 con 0x7f2be809ac40 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2be81d8100 con 0x7f2be809ac40 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2bd8001860 con 0x7f2be809ac40 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.673 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1218689457 0 0) 0x7f2be811f8d0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 >> v1:172.21.15.100:6790/0 conn(0x7f2be809ac40 legacy=0x7f2be80afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 >> v1:172.21.15.100:6789/0 conn(0x7f2be80b07a0 legacy=0x7f2be81d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 monclient: found mon.b 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be81e64c0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2bee6ba640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2bee6ba640 10 monclient: _renew_subs 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2bee6ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be81e3490 con 0x7f2be81d7cd0 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2bee6ba640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2bee6ba640 10 monclient: _renew_subs 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2bee6ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be81e39c0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2bee6ba640 1 librados: init done 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2be00020d0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2be0003b30 con 0x7f2be81d7cd0 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 10 monclient: dump: 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be67fc640 4 set_mon_vals no callback set 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.658+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2be00040a0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.659+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 4229708925 0 0) 0x7f2be0006a80 con 0x7f2be81d7cd0 2024-03-29T11:36:25.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.659+0000 7f2be4ff9640 10 client.4147.objecter ms_dispatch 0x7f2be8000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:36:25.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.659+0000 7f2be4ff9640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:36:25.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.659+0000 7f2be4ff9640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:36:25.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.659+0000 7f2be4ff9640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:36:25.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.660+0000 7f2bee6ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.660+0000 7f2bee6ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.660+0000 7f2bee6ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.660+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be81d72e0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.662+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2be81d72e0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.662+0000 7f2be4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.662+0000 7f2be4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:25.744 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.745+0000 7f2bee6ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fee1d4d7-1159-4388-ad5f-82e515d74b23", "id": 5}] 2024-03-29T11:36:25.744 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.745+0000 7f2bee6ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fee1d4d7-1159-4388-ad5f-82e515d74b23", "id": 5}] 2024-03-29T11:36:25.744 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.745+0000 7f2bee6ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.744 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.745+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3930222204 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fee1d4d7-1159-4388-ad5f-82e515d74b23", "id": 5} v 0) v1 -- 0x7f2be80b0be0 con 0x7f2be81d7cd0 2024-03-29T11:36:25.749 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.750+0000 7f2be4ff9640 1 -- 172.21.15.191:0/3930222204 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fee1d4d7-1159-4388-ad5f-82e515d74b23", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2165078005 0 1988412176) 0x7f2be0005d30 con 0x7f2be81d7cd0 2024-03-29T11:36:25.749 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.750+0000 7f2be4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fee1d4d7-1159-4388-ad5f-82e515d74b23", "id": 5}] 2024-03-29T11:36:25.749 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.750+0000 7f2be4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:25.749 INFO:teuthology.orchestra.run.smithi191.stdout:5 2024-03-29T11:36:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 10 librados: watch_flush enter 2024-03-29T11:36:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 10 librados: watch_flush exit 2024-03-29T11:36:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:36:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:36:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 10 monclient: shutdown 2024-03-29T11:36:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3930222204 >> v1:172.21.15.191:6789/0 conn(0x7f2be81d7cd0 legacy=0x7f2be81e1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3930222204 shutdown_connections 2024-03-29T11:36:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3930222204 >> 172.21.15.191:0/3930222204 conn(0x7f2be8066b00 msgr2=0x7f2be80b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3930222204 shutdown_connections 2024-03-29T11:36:25.752 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 1 -- 172.21.15.191:0/3930222204 wait complete. 2024-03-29T11:36:25.752 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.751+0000 7f2bee6ba640 1 librados: shutdown 2024-03-29T11:36:25.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 559b1250-e1fa-43cf-b727-d47eae0a4a54 6 2024-03-29T11:36:25.861 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.862+0000 7f5591e4a640 10 monclient: get_monmap_and_config 2024-03-29T11:36:25.861 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.862+0000 7f5591e4a640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.861 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.862+0000 7f5591e4a640 10 monclient: monmap: 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:36:25.863104+0000 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:36:25.863104+0000 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.862+0000 7f5591e4a640 1 Processor -- start 2024-03-29T11:36:25.862 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.862+0000 7f5591e4a640 1 -- start start 2024-03-29T11:36:25.863 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.862+0000 7f5591e4a640 10 monclient: init 2024-03-29T11:36:25.872 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.863+0000 7f5591e4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.863+0000 7f5591e4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:25.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.863+0000 7f5591e4a640 10 monclient(hunting): picked mon.noname-b con 0x7f558c09ac40 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:25.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.863+0000 7f5591e4a640 10 monclient(hunting): picked mon.noname-a con 0x7f558c09c870 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:25.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.863+0000 7f5591e4a640 10 monclient(hunting): picked mon.noname-c con 0x7f558c0b4360 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:25.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.863+0000 7f5591e4a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c09dec0 con 0x7f558c09c870 2024-03-29T11:36:25.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f5591e4a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c09da30 con 0x7f558c0b4360 2024-03-29T11:36:25.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558bfff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f558c0b4360 0x7f558c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49924/0 (socket says 172.21.15.191:49924) 2024-03-29T11:36:25.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f5591e4a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c0b39c0 con 0x7f558c09ac40 2024-03-29T11:36:25.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558affd640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f558c09c870 0x7f558c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.191:58734/0 (socket says 172.21.15.191:58734) 2024-03-29T11:36:25.873 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f5591e4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558bfff640 1 -- 172.21.15.191:0/1032293183 learned_addr learned my addr 172.21.15.191:0/1032293183 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558b7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f558c09ac40 0x7f558c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.191:56428/0 (socket says 172.21.15.191:56428) 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f5591e4a640 10 monclient(hunting): authenticate will time out at 896.181337s 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785769219 0 0) 0x7f558c09da30 con 0x7f558c0b4360 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5574003520 con 0x7f558c0b4360 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164146203 0 0) 0x7f558c0b39c0 con 0x7f558c09ac40 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.874 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f558c09da30 con 0x7f558c09ac40 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070763050 0 0) 0x7f5580001110 con 0x7f558c09c870 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f558c0b39c0 con 0x7f558c09c870 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 711508565 0 0) 0x7f5574003520 con 0x7f558c0b4360 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 >> v1:172.21.15.100:6790/0 conn(0x7f558c09ac40 legacy=0x7f558c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.875 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 >> v1:172.21.15.100:6789/0 conn(0x7f558c09c870 legacy=0x7f558c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c0b5160 con 0x7f558c0b4360 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5584001690 con 0x7f558c0b4360 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f5591e4a640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:36:25.876 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f5591e4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: dump: 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.877 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f5591e4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5584001870 con 0x7f558c0b4360 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 1 -- 172.21.15.191:0/1032293183 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5584001cc0 con 0x7f558c0b4360 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f5591e4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f5591e4a640 4 set_mon_vals no callback set 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.864+0000 7f558a7fc640 10 monclient: dump: 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.878 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.865+0000 7f5591e4a640 10 monclient: shutdown 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.865+0000 7f5591e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.865+0000 7f5591e4a640 1 -- 172.21.15.191:0/1032293183 >> v1:172.21.15.191:6789/0 conn(0x7f558c0b4360 legacy=0x7f558c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.879 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.865+0000 7f5591e4a640 1 -- 172.21.15.191:0/1032293183 shutdown_connections 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.865+0000 7f5591e4a640 1 -- 172.21.15.191:0/1032293183 >> 172.21.15.191:0/1032293183 conn(0x7f558c066b00 msgr2=0x7f558c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.865+0000 7f5591e4a640 1 -- 172.21.15.191:0/1032293183 shutdown_connections 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.865+0000 7f5591e4a640 1 -- 172.21.15.191:0/1032293183 wait complete. 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.866+0000 7f5591e4a640 10 monclient: build_initial_monmap 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.866+0000 7f5591e4a640 10 monclient: monmap: 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.880 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.866+0000 7f5591e4a640 1 librados: starting msgr at 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.866+0000 7f5591e4a640 1 librados: starting objecter 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.866+0000 7f5591e4a640 1 Processor -- start 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.866+0000 7f5591e4a640 1 -- start start 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.866+0000 7f5591e4a640 1 librados: setting wanted keys 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.866+0000 7f5591e4a640 1 librados: calling monclient init 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.866+0000 7f5591e4a640 10 monclient: init 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5591e4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:25.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5591e4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5591e4a640 10 monclient(hunting): picked mon.noname-c con 0x7f558c09ac40 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5591e4a640 10 monclient(hunting): picked mon.noname-a con 0x7f558c1d7580 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5591e4a640 10 monclient(hunting): picked mon.noname-b con 0x7f558c1dfb50 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5591e4a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c13ffa0 con 0x7f558c1d7580 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5591e4a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c1d7060 con 0x7f558c1dfb50 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5591e4a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c0adb70 con 0x7f558c09ac40 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5591e4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5591e4a640 10 monclient(hunting): authenticate will time out at 896.184488s 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f558bfff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f558c1dfb50 0x7f558c1e1f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49940/0 (socket says 172.21.15.191:49940) 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f558bfff640 1 -- 172.21.15.191:0/1238999689 learned_addr learned my addr 172.21.15.191:0/1238999689 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 client.?.objecter ms_handle_connect 0x7f558c1dfb50 2024-03-29T11:36:25.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 client.?.objecter ms_handle_connect 0x7f558c09ac40 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 client.?.objecter ms_handle_connect 0x7f558c1d7580 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902350967 0 0) 0x7f558c1d7060 con 0x7f558c1dfb50 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5578004eb0 con 0x7f558c1dfb50 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437373399 0 0) 0x7f558c0adb70 con 0x7f558c09ac40 2024-03-29T11:36:25.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f558c1d7060 con 0x7f558c09ac40 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397067517 0 0) 0x7f558c13ffa0 con 0x7f558c1d7580 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f558c0adb70 con 0x7f558c1d7580 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.867+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 858039118 0 0) 0x7f5578004eb0 con 0x7f558c1dfb50 2024-03-29T11:36:25.884 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f558c13ffa0 con 0x7f558c1dfb50 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5584001bc0 con 0x7f558c1dfb50 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient(hunting): dump: 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1277791148 0 0) 0x7f558c1d7060 con 0x7f558c09ac40 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5578004eb0 con 0x7f558c09ac40 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 168301532 0 0) 0x7f558c0adb70 con 0x7f558c1d7580 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f558c1d7060 con 0x7f558c1d7580 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f557c001de0 con 0x7f558c09ac40 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient(hunting): dump: 2024-03-29T11:36:25.886 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5580001760 con 0x7f558c1d7580 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:25.887 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient(hunting): dump: 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 527372052 0 0) 0x7f558c13ffa0 con 0x7f558c1dfb50 2024-03-29T11:36:25.888 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 >> v1:172.21.15.100:6790/0 conn(0x7f558c09ac40 legacy=0x7f558c0b0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 >> v1:172.21.15.100:6789/0 conn(0x7f558c1d7580 legacy=0x7f558c1d79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 monclient: found mon.b 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c0b16e0 con 0x7f558c1dfb50 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5591e4a640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5591e4a640 10 monclient: _renew_subs 2024-03-29T11:36:25.889 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5591e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5591e4a640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558c0b0430 con 0x7f558c1dfb50 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5591e4a640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5591e4a640 10 monclient: _renew_subs 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5591e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5591e4a640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c1e3840 con 0x7f558c1dfb50 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5591e4a640 1 librados: init done 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5584004a90 con 0x7f558c1dfb50 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5584004da0 con 0x7f558c1dfb50 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:25.890 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5589ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 monclient: dump: 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:25.891 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5584005290 con 0x7f558c1dfb50 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1341294086 0 0) 0x7f5584006a10 con 0x7f558c1dfb50 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 10 client.4153.objecter ms_dispatch 0x7f558c000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.868+0000 7f5573fff640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.869+0000 7f5591e4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.869+0000 7f5591e4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.869+0000 7f5591e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.869+0000 7f5591e4a640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c005080 con 0x7f558c1dfb50 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.871+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f55840064d0 con 0x7f558c1dfb50 2024-03-29T11:36:25.892 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.871+0000 7f5573fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:25.893 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.871+0000 7f5573fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:25.942 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.943+0000 7f5591e4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "559b1250-e1fa-43cf-b727-d47eae0a4a54", "id": 6}] 2024-03-29T11:36:25.942 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.943+0000 7f5591e4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "559b1250-e1fa-43cf-b727-d47eae0a4a54", "id": 6}] 2024-03-29T11:36:25.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.943+0000 7f5591e4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:25.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.943+0000 7f5591e4a640 1 -- 172.21.15.191:0/1238999689 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd new", "uuid": "559b1250-e1fa-43cf-b727-d47eae0a4a54", "id": 6} v 0) v1 -- 0x7f556c0058b0 con 0x7f558c1dfb50 2024-03-29T11:36:25.948 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.949+0000 7f5573fff640 1 -- 172.21.15.191:0/1238999689 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "559b1250-e1fa-43cf-b727-d47eae0a4a54", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 4282504310 0 1113811849) 0x7f5584006740 con 0x7f558c1dfb50 2024-03-29T11:36:25.948 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.949+0000 7f5573fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "559b1250-e1fa-43cf-b727-d47eae0a4a54", "id": 6}] 2024-03-29T11:36:25.948 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.949+0000 7f5573fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:25.948 INFO:teuthology.orchestra.run.smithi191.stdout:6 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 10 librados: watch_flush enter 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 10 librados: watch_flush exit 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 10 client.4153.objecter successfully canceled tick 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 10 monclient: shutdown 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 1 -- 172.21.15.191:0/1238999689 >> v1:172.21.15.191:6789/0 conn(0x7f558c1dfb50 legacy=0x7f558c1e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 1 -- 172.21.15.191:0/1238999689 shutdown_connections 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 1 -- 172.21.15.191:0/1238999689 >> 172.21.15.191:0/1238999689 conn(0x7f558c066b00 msgr2=0x7f558c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 1 -- 172.21.15.191:0/1238999689 shutdown_connections 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.950+0000 7f5591e4a640 1 -- 172.21.15.191:0/1238999689 wait complete. 2024-03-29T11:36:25.950 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:25.951+0000 7f5591e4a640 1 librados: shutdown 2024-03-29T11:36:25.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 23c4000f-1f04-48ef-a47c-9b8d2035383d 7 2024-03-29T11:36:25.999 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:26.000+0000 7f1b76fdb040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:36:26.015 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:26.016+0000 7fda32e32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:36:26.060 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.061+0000 7f0b5df9d640 10 monclient: get_monmap_and_config 2024-03-29T11:36:26.060 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.061+0000 7f0b5df9d640 10 monclient: build_initial_monmap 2024-03-29T11:36:26.060 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.061+0000 7f0b5df9d640 10 monclient: monmap: 2024-03-29T11:36:26.060 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:26.060 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.060 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:36:26.062244+0000 2024-03-29T11:36:26.060 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:36:26.062244+0000 2024-03-29T11:36:26.060 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:26.060 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:26.061 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:26.061 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:26.061 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:26.061 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 1 Processor -- start 2024-03-29T11:36:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 1 -- start start 2024-03-29T11:36:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 10 monclient: init 2024-03-29T11:36:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:26.069 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b580b1780 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b5813d7e0 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0b58143be0 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b58070b30 con 0x7f0b580b1780 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b58070da0 con 0x7f0b58143be0 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5809adb0 con 0x7f0b5813d7e0 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b5df9d640 10 monclient(hunting): authenticate will time out at 896.379812s 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b57fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f0b58143be0 0x7f0b5814b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49956/0 (socket says 172.21.15.191:49956) 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.062+0000 7f0b57fff640 1 -- 172.21.15.191:0/1110431805 learned_addr learned my addr 172.21.15.191:0/1110431805 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422461081 0 0) 0x7f0b58070da0 con 0x7f0b58143be0 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b40003520 con 0x7f0b58143be0 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766625749 0 0) 0x7f0b58070b30 con 0x7f0b580b1780 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b58070da0 con 0x7f0b580b1780 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002738980 0 0) 0x7f0b5809adb0 con 0x7f0b5813d7e0 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:36:26.071 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b58070b30 con 0x7f0b5813d7e0 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1636645296 0 0) 0x7f0b40003520 con 0x7f0b58143be0 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 >> v1:172.21.15.100:6790/0 conn(0x7f0b5813d7e0 legacy=0x7f0b581434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 >> v1:172.21.15.100:6789/0 conn(0x7f0b580b1780 legacy=0x7f0b5813d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 monclient: found mon.noname-c 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b58144070 con 0x7f0b58143be0 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:26.072 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0b50001a10 con 0x7f0b58143be0 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b5df9d640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b5df9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: dump: 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.073 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b5df9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b50001bf0 con 0x7f0b58143be0 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 1 -- 172.21.15.191:0/1110431805 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0b50002000 con 0x7f0b58143be0 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b5df9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b5df9d640 4 set_mon_vals no callback set 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.063+0000 7f0b567fc640 10 monclient: dump: 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.074 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.075 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.075 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:26.075 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.075 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.075 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.075 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:26.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.064+0000 7f0b5df9d640 10 monclient: shutdown 2024-03-29T11:36:26.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.064+0000 7f0b5df9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:26.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.064+0000 7f0b5df9d640 1 -- 172.21.15.191:0/1110431805 >> v1:172.21.15.191:6789/0 conn(0x7f0b58143be0 legacy=0x7f0b5814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.075 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.064+0000 7f0b5df9d640 1 -- 172.21.15.191:0/1110431805 shutdown_connections 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.064+0000 7f0b5df9d640 1 -- 172.21.15.191:0/1110431805 >> 172.21.15.191:0/1110431805 conn(0x7f0b580b58f0 msgr2=0x7f0b580b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.064+0000 7f0b5df9d640 1 -- 172.21.15.191:0/1110431805 shutdown_connections 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.064+0000 7f0b5df9d640 1 -- 172.21.15.191:0/1110431805 wait complete. 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.064+0000 7f0b5df9d640 10 monclient: build_initial_monmap 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.064+0000 7f0b5df9d640 10 monclient: monmap: 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:26.076 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 1 librados: starting msgr at 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 1 librados: starting objecter 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 1 Processor -- start 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 1 -- start start 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 1 librados: setting wanted keys 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 1 librados: calling monclient init 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 10 monclient: init 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0b580b1780 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:26.077 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b5813d7e0 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b58143be0 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b580ab260 con 0x7f0b5813d7e0 2024-03-29T11:36:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5813ff80 con 0x7f0b58143be0 2024-03-29T11:36:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5813fca0 con 0x7f0b580b1780 2024-03-29T11:36:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.065+0000 7f0b5df9d640 10 monclient(hunting): authenticate will time out at 896.383060s 2024-03-29T11:36:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b57fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f0b58143be0 0x7f0b581df710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49972/0 (socket says 172.21.15.191:49972) 2024-03-29T11:36:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b57fff640 1 -- 172.21.15.191:0/2910029191 learned_addr learned my addr 172.21.15.191:0/2910029191 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-29T11:36:26.078 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 client.?.objecter ms_handle_connect 0x7f0b58143be0 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 client.?.objecter ms_handle_connect 0x7f0b5813d7e0 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 client.?.objecter ms_handle_connect 0x7f0b580b1780 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812094272 0 0) 0x7f0b5813ff80 con 0x7f0b58143be0 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): my global_id is 4159 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b44004eb0 con 0x7f0b58143be0 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272302348 0 0) 0x7f0b580ab260 con 0x7f0b5813d7e0 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.079 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b5813ff80 con 0x7f0b5813d7e0 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850020492 0 0) 0x7f0b5813fca0 con 0x7f0b580b1780 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b580ab260 con 0x7f0b580b1780 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1667439925 0 0) 0x7f0b44004eb0 con 0x7f0b58143be0 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b5813fca0 con 0x7f0b58143be0 2024-03-29T11:36:26.080 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 689270213 0 0) 0x7f0b5813ff80 con 0x7f0b5813d7e0 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b44004eb0 con 0x7f0b5813d7e0 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0b50001f40 con 0x7f0b58143be0 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): dump: 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.081 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0b4c001870 con 0x7f0b5813d7e0 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): dump: 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.082 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.083 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:26.083 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.083 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.083 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.083 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:26.083 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 980669001 0 0) 0x7f0b580ab260 con 0x7f0b580b1780 2024-03-29T11:36:26.083 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b5813ff80 con 0x7f0b580b1780 2024-03-29T11:36:26.083 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0b48001710 con 0x7f0b580b1780 2024-03-29T11:36:26.083 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient(hunting): dump: 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.084 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1030338202 0 0) 0x7f0b5813fca0 con 0x7f0b58143be0 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 >> v1:172.21.15.100:6790/0 conn(0x7f0b580b1780 legacy=0x7f0b5813f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 >> v1:172.21.15.100:6789/0 conn(0x7f0b5813d7e0 legacy=0x7f0b581d4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 monclient: found mon.b 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b581404a0 con 0x7f0b58143be0 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient: _finish_auth 0 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b5df9d640 5 monclient: authenticate success, global_id 4159 2024-03-29T11:36:26.085 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b5df9d640 10 monclient: _renew_subs 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b5df9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b5df9d640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b58142070 con 0x7f0b58143be0 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b5df9d640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b5df9d640 10 monclient: _renew_subs 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b5df9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b5df9d640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b581d5330 con 0x7f0b58143be0 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b5df9d640 1 librados: init done 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b500020a0 con 0x7f0b58143be0 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0b50004fb0 con 0x7f0b58143be0 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.086 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b55ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 monclient: dump: 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:36:26.087 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0b500054a0 con 0x7f0b58143be0 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1778091283 0 0) 0x7f0b50006d30 con 0x7f0b58143be0 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.066+0000 7f0b3ffff640 10 client.4159.objecter ms_dispatch 0x7f0b58000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.067+0000 7f0b3ffff640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.067+0000 7f0b3ffff640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.067+0000 7f0b3ffff640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.067+0000 7f0b5df9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.067+0000 7f0b5df9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.067+0000 7f0b5df9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.067+0000 7f0b5df9d640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b38005080 con 0x7f0b58143be0 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.069+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0b500066d0 con 0x7f0b58143be0 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.069+0000 7f0b3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:26.088 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.069+0000 7f0b3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:26.134 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:26.135+0000 7f1b76fdb040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:36:26.141 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.142+0000 7f0b5df9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "23c4000f-1f04-48ef-a47c-9b8d2035383d", "id": 7}] 2024-03-29T11:36:26.141 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.142+0000 7f0b5df9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "23c4000f-1f04-48ef-a47c-9b8d2035383d", "id": 7}] 2024-03-29T11:36:26.141 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.142+0000 7f0b5df9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:26.141 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.142+0000 7f0b5df9d640 1 -- 172.21.15.191:0/2910029191 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd new", "uuid": "23c4000f-1f04-48ef-a47c-9b8d2035383d", "id": 7} v 0) v1 -- 0x7f0b380058b0 con 0x7f0b58143be0 2024-03-29T11:36:26.144 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.145+0000 7f0b3ffff640 1 -- 172.21.15.191:0/2910029191 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "23c4000f-1f04-48ef-a47c-9b8d2035383d", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3092288837 0 1371665406) 0x7f0b50006a00 con 0x7f0b58143be0 2024-03-29T11:36:26.144 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.145+0000 7f0b3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "23c4000f-1f04-48ef-a47c-9b8d2035383d", "id": 7}] 2024-03-29T11:36:26.144 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.145+0000 7f0b3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:26.144 INFO:teuthology.orchestra.run.smithi191.stdout:7 2024-03-29T11:36:26.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 10 librados: watch_flush enter 2024-03-29T11:36:26.145 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 10 librados: watch_flush exit 2024-03-29T11:36:26.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-29T11:36:26.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 10 client.4159.objecter successfully canceled tick 2024-03-29T11:36:26.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 10 monclient: shutdown 2024-03-29T11:36:26.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:26.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 1 -- 172.21.15.191:0/2910029191 >> v1:172.21.15.191:6789/0 conn(0x7f0b58143be0 legacy=0x7f0b581df710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 1 -- 172.21.15.191:0/2910029191 shutdown_connections 2024-03-29T11:36:26.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 1 -- 172.21.15.191:0/2910029191 >> 172.21.15.191:0/2910029191 conn(0x7f0b580b58f0 msgr2=0x7f0b58069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:26.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 1 -- 172.21.15.191:0/2910029191 shutdown_connections 2024-03-29T11:36:26.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 1 -- 172.21.15.191:0/2910029191 wait complete. 2024-03-29T11:36:26.146 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-29T11:36:26.146+0000 7f0b5df9d640 1 librados: shutdown 2024-03-29T11:36:26.157 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:36:26.157 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:36:26.158 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:26.157+0000 7fda32e32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:36:26.160 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:36:26.160 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:36:26.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:36:26.162 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:36:26.162 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:36:26.162 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:36:26.164 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:36:26.164 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:36:26.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:36:26.166 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:36:26.166 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:36:26.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:36:26.168 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:36:26.169 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:36:26.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:36:26.171 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:36:26.171 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:36:26.171 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:36:26.173 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:36:26.173 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:36:26.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:36:26.175 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:36:26.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:26.284 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:26.285+0000 7f1b76fdb040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:36:26.296 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:26.298+0000 7fda32e32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:36:26.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.302+0000 7fb85d6ae640 10 monclient: get_monmap_and_config 2024-03-29T11:36:26.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.302+0000 7fb85d6ae640 10 monclient: build_initial_monmap 2024-03-29T11:36:26.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.302+0000 7fb85d6ae640 10 monclient: monmap: 2024-03-29T11:36:26.301 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-03-29T11:36:26.301 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.301 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:36:26.303385+0000 2024-03-29T11:36:26.301 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:36:26.303385+0000 2024-03-29T11:36:26.301 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:26.301 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:26.302 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:26.302 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:26.302 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:26.302 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:26.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.302+0000 7fb85d6ae640 1 Processor -- start 2024-03-29T11:36:26.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.302+0000 7fb85d6ae640 1 -- start start 2024-03-29T11:36:26.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 10 monclient: init 2024-03-29T11:36:26.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:26.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:26.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 10 monclient(hunting): picked mon.noname-c con 0x7fb85806ee90 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:26.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 10 monclient(hunting): picked mon.noname-b con 0x7fb85806f270 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:26.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 10 monclient(hunting): picked mon.noname-a con 0x7fb8581423d0 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:26.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85806fcc0 con 0x7fb8581423d0 2024-03-29T11:36:26.312 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85806ffa0 con 0x7fb85806ee90 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb8567fc640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fb85806f270 0x7fb858141cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:55008/0 (socket says 172.21.15.100:55008) 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb8567fc640 1 -- 172.21.15.100:0/685063303 learned_addr learned my addr 172.21.15.100:0/685063303 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 1 -- 172.21.15.100:0/685063303 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb858071150 con 0x7fb85806f270 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.303+0000 7fb85d6ae640 10 monclient(hunting): authenticate will time out at 906.498964s 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828114544 0 0) 0x7fb85806fcc0 con 0x7fb8581423d0 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb840004630 con 0x7fb8581423d0 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761861128 0 0) 0x7fb858071150 con 0x7fb85806f270 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.313 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): my global_id is 4178 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85806fcc0 con 0x7fb85806f270 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197324945 0 0) 0x7fb85806ffa0 con 0x7fb85806ee90 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb858071150 con 0x7fb85806ee90 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2478623782 0 0) 0x7fb85806fcc0 con 0x7fb85806f270 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 >> v1:172.21.15.191:6789/0 conn(0x7fb85806ee90 legacy=0x7fb85813b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 >> v1:172.21.15.100:6789/0 conn(0x7fb8581423d0 legacy=0x7fb858149d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 monclient: found mon.noname-b 2024-03-29T11:36:26.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8580713c0 con 0x7fb85806f270 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb84c004fc0 con 0x7fb85806f270 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb85d6ae640 5 monclient: authenticate success, global_id 4178 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb85d6ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: dump: 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.315 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb85d6ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb84c000ed0 con 0x7fb85806f270 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 1 -- 172.21.15.100:0/685063303 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb84c0012a0 con 0x7fb85806f270 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb85d6ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb85d6ae640 4 set_mon_vals no callback set 2024-03-29T11:36:26.316 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.304+0000 7fb855ffb640 10 monclient: dump: 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.305+0000 7fb85d6ae640 10 monclient: shutdown 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.305+0000 7fb85d6ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.305+0000 7fb85d6ae640 1 -- 172.21.15.100:0/685063303 >> v1:172.21.15.100:6790/0 conn(0x7fb85806f270 legacy=0x7fb858141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.317 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.305+0000 7fb85d6ae640 1 -- 172.21.15.100:0/685063303 shutdown_connections 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.305+0000 7fb85d6ae640 1 -- 172.21.15.100:0/685063303 >> 172.21.15.100:0/685063303 conn(0x7fb8580b41f0 msgr2=0x7fb8580b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.305+0000 7fb85d6ae640 1 -- 172.21.15.100:0/685063303 shutdown_connections 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.305+0000 7fb85d6ae640 1 -- 172.21.15.100:0/685063303 wait complete. 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.305+0000 7fb85d6ae640 10 monclient: build_initial_monmap 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.305+0000 7fb85d6ae640 10 monclient: monmap: 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 0 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 0.000000 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:created 0.000000 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:26.318 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 1 librados: starting msgr at 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 1 librados: starting objecter 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 1 Processor -- start 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 1 -- start start 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 1 librados: setting wanted keys 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 1 librados: calling monclient init 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 10 monclient: init 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 10 monclient(hunting): picked mon.noname-a con 0x7fb85806ee90 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.306+0000 7fb85d6ae640 10 monclient(hunting): picked mon.noname-b con 0x7fb85806f270 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 10 monclient(hunting): picked mon.noname-c con 0x7fb8581423d0 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb856ffd640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fb85806ee90 0x7fb8581da770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:34178/0 (socket says 172.21.15.100:34178) 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb856ffd640 1 -- 172.21.15.100:0/68082437 learned_addr learned my addr 172.21.15.100:0/68082437 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:26.319 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 client.?.objecter ms_handle_connect 0x7fb85806ee90 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 client.?.objecter ms_handle_connect 0x7fb85806f270 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 client.?.objecter ms_handle_connect 0x7fb8581423d0 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb858142980 con 0x7fb85806ee90 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85812ec30 con 0x7fb85806f270 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb858137120 con 0x7fb8581423d0 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870897188 0 0) 0x7fb858142980 con 0x7fb85806ee90 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 10 monclient(hunting): authenticate will time out at 906.502537s 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.320 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb844004eb0 con 0x7fb85806ee90 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772920887 0 0) 0x7fb85812ec30 con 0x7fb85806f270 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb858142980 con 0x7fb85806f270 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902564860 0 0) 0x7fb858137120 con 0x7fb8581423d0 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85812ec30 con 0x7fb8581423d0 2024-03-29T11:36:26.321 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1044192961 0 0) 0x7fb844004eb0 con 0x7fb85806ee90 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb858137120 con 0x7fb85806ee90 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb848001970 con 0x7fb85806ee90 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): dump: 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:26.322 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2766060011 0 0) 0x7fb858142980 con 0x7fb85806f270 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb844004eb0 con 0x7fb85806f270 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb84c0016c0 con 0x7fb85806f270 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): dump: 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.323 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1326717123 0 0) 0x7fb85812ec30 con 0x7fb8581423d0 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb858142980 con 0x7fb8581423d0 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb850001410 con 0x7fb8581423d0 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient(hunting): dump: 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.324 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3403286138 0 0) 0x7fb858137120 con 0x7fb85806ee90 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 >> v1:172.21.15.100:6790/0 conn(0x7fb8581423d0 legacy=0x7fb85812e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 >> v1:172.21.15.191:6789/0 conn(0x7fb85806f270 legacy=0x7fb85812de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 monclient: found mon.a 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8581379b0 con 0x7fb85806ee90 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb83f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:36:26.325 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 10 monclient: _renew_subs 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb858141670 con 0x7fb85806ee90 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 10 monclient: _renew_subs 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8581db060 con 0x7fb85806ee90 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.307+0000 7fb85d6ae640 1 librados: init done 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb848001c50 con 0x7fb85806ee90 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb848003a60 con 0x7fb85806ee90 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb8557fa640 4 set_mon_vals no callback set 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 10 monclient: dump: 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:epoch 1 2024-03-29T11:36:26.326 INFO:teuthology.orchestra.run.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb848004010 con 0x7fb85806ee90 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fb848006980 con 0x7fb85806ee90 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 10 client.4188.objecter ms_dispatch 0x7fb858000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb85d6ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:26.327 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.308+0000 7fb83f7fe640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-29T11:36:26.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.309+0000 7fb85d6ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:26.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.309+0000 7fb85d6ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:26.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.309+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb838005080 con 0x7fb85806ee90 2024-03-29T11:36:26.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.311+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb848006b90 con 0x7fb85806ee90 2024-03-29T11:36:26.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.311+0000 7fb83f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:26.328 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.311+0000 7fb83f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:26.332 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:26.334+0000 7f1b76fdb040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:36:26.341 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:26.343+0000 7fda32e32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:36:26.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.372+0000 7fb85d6ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:26.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.372+0000 7fb85d6ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:26.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.372+0000 7fb85d6ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:26.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.372+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb838005330 con 0x7fb85806ee90 2024-03-29T11:36:26.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.372+0000 7fb83f7fe640 1 -- 172.21.15.100:0/68082437 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fb848006270 con 0x7fb85806ee90 2024-03-29T11:36:26.371 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.372+0000 7fb83f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:26.372 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.372+0000 7fb83f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:26.372 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-29T11:36:26.372 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:26.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 10 librados: watch_flush enter 2024-03-29T11:36:26.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 10 librados: watch_flush exit 2024-03-29T11:36:26.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-29T11:36:26.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 10 client.4188.objecter successfully canceled tick 2024-03-29T11:36:26.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 10 monclient: shutdown 2024-03-29T11:36:26.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:26.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 >> v1:172.21.15.100:6789/0 conn(0x7fb85806ee90 legacy=0x7fb8581da770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:26.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 shutdown_connections 2024-03-29T11:36:26.373 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 >> 172.21.15.100:0/68082437 conn(0x7fb8580b41f0 msgr2=0x7fb85813b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:26.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 shutdown_connections 2024-03-29T11:36:26.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 1 -- 172.21.15.100:0/68082437 wait complete. 2024-03-29T11:36:26.374 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-29T11:36:26.373+0000 7fb85d6ae640 1 librados: shutdown 2024-03-29T11:36:26.383 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:36:26.383 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:36:26.448 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:26.449+0000 7f1b76fdb040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:36:26.452 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:26.453+0000 7fda32e32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:36:26.544 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:26.545+0000 7f1b76fdb040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:36:26.547 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:26.548+0000 7fda32e32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:36:26.698 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:26.699+0000 7fda32e32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:36:26.699 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:26.699+0000 7f1b76fdb040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:36:26.740 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:26.741+0000 7fda32e32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:36:26.740 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:26.742+0000 7f1b76fdb040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:36:26.774 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-29T11:36:26.775+0000 7f979fb73640 -1 Falling back to public interface 2024-03-29T11:36:26.776 INFO:tasks.ceph.osd.0.smithi100.stderr:2024-03-29T11:36:26.767+0000 7fb77ad14640 -1 Falling back to public interface 2024-03-29T11:36:26.776 INFO:tasks.ceph.osd.2.smithi100.stderr:2024-03-29T11:36:26.771+0000 7fabb7e0d640 -1 Falling back to public interface 2024-03-29T11:36:26.777 INFO:tasks.ceph.osd.3.smithi100.stderr:2024-03-29T11:36:26.776+0000 7f84138d7640 -1 Falling back to public interface 2024-03-29T11:36:26.778 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-29T11:36:26.779+0000 7f7ef0d24640 -1 Falling back to public interface 2024-03-29T11:36:26.787 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-03-29T11:36:26.788+0000 7f6e4c068640 -1 Falling back to public interface 2024-03-29T11:36:26.795 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-29T11:36:26.793+0000 7fc289a10640 -1 Falling back to public interface 2024-03-29T11:36:26.797 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-29T11:36:26.799+0000 7f82c8781640 -1 Falling back to public interface 2024-03-29T11:36:26.898 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:26.899+0000 7fda32e32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:36:26.899 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:26.899+0000 7f1b76fdb040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:36:27.189 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:27.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.280+0000 7f1b2b8d9640 10 monclient: get_monmap_and_config 2024-03-29T11:36:27.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.280+0000 7f1b2b8d9640 10 monclient: build_initial_monmap 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.280+0000 7f1b2b8d9640 10 monclient: monmap: 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:36:27.281614+0000 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:36:27.281614+0000 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:27.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 1 Processor -- start 2024-03-29T11:36:27.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 1 -- start start 2024-03-29T11:36:27.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 10 monclient: init 2024-03-29T11:36:27.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:27.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:27.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 10 monclient(hunting): picked mon.noname-a con 0x7f1b241457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:27.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 10 monclient(hunting): picked mon.noname-c con 0x7f1b2413f650 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:27.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 10 monclient(hunting): picked mon.noname-b con 0x7f1b240a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:27.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b240aaef0 con 0x7f1b241457f0 2024-03-29T11:36:27.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b24145bd0 con 0x7f1b2413f650 2024-03-29T11:36:27.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b24145eb0 con 0x7f1b240a9d70 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.281+0000 7f1b2b8d9640 10 monclient(hunting): authenticate will time out at 907.477019s 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b29675640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f1b241457f0 0x7f1b2413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:34180/0 (socket says 172.21.15.100:34180) 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b29675640 1 -- 172.21.15.100:0/2375429212 learned_addr learned my addr 172.21.15.100:0/2375429212 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753280140 0 0) 0x7f1b240aaef0 con 0x7f1b241457f0 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b10003520 con 0x7f1b241457f0 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257316652 0 0) 0x7f1b24145eb0 con 0x7f1b240a9d70 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:27.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:36:27.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b240aaef0 con 0x7f1b240a9d70 2024-03-29T11:36:27.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352174640 0 0) 0x7f1b24145bd0 con 0x7f1b2413f650 2024-03-29T11:36:27.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:27.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:27.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:27.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:36:27.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24145eb0 con 0x7f1b2413f650 2024-03-29T11:36:27.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3873937446 0 0) 0x7f1b10003520 con 0x7f1b241457f0 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 >> v1:172.21.15.100:6790/0 conn(0x7f1b240a9d70 legacy=0x7f1b2414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 >> v1:172.21.15.191:6789/0 conn(0x7f1b2413f650 legacy=0x7f1b240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 monclient: found mon.noname-a 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b24146340 con 0x7f1b241457f0 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1b20001690 con 0x7f1b241457f0 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:27.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: dump: 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b2b8d9640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b2b8d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b20001870 con 0x7f1b241457f0 2024-03-29T11:36:27.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 1 -- 172.21.15.100:0/2375429212 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1b20001cc0 con 0x7f1b241457f0 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b2b8d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b2b8d9640 4 set_mon_vals no callback set 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.282+0000 7f1b1bfff640 10 monclient: dump: 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:27.294 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.283+0000 7f1b2b8d9640 10 monclient: shutdown 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.283+0000 7f1b2b8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.283+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/2375429212 >> v1:172.21.15.100:6789/0 conn(0x7f1b241457f0 legacy=0x7f1b2413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.283+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/2375429212 shutdown_connections 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.283+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/2375429212 >> 172.21.15.100:0/2375429212 conn(0x7f1b24066a00 msgr2=0x7f1b24066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.283+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/2375429212 shutdown_connections 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.283+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/2375429212 wait complete. 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.283+0000 7f1b2b8d9640 10 monclient: build_initial_monmap 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.283+0000 7f1b2b8d9640 10 monclient: monmap: 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:36:27.295 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 1 librados: starting msgr at 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 1 librados: starting objecter 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 1 Processor -- start 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 1 -- start start 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 1 librados: setting wanted keys 2024-03-29T11:36:27.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 1 librados: calling monclient init 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 10 monclient: init 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 10 monclient(hunting): picked mon.noname-b con 0x7f1b240a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 10 monclient(hunting): picked mon.noname-c con 0x7f1b2413f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 10 monclient(hunting): picked mon.noname-a con 0x7f1b241457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b240af1e0 con 0x7f1b241457f0 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b240ae340 con 0x7f1b240a9d70 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b240ad340 con 0x7f1b2413f650 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.284+0000 7f1b2b8d9640 10 monclient(hunting): authenticate will time out at 907.480008s 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b28e74640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f1b2413f650 0x7f1b24139cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:55042/0 (socket says 172.21.15.100:55042) 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b28e74640 1 -- 172.21.15.100:0/3855842613 learned_addr learned my addr 172.21.15.100:0/3855842613 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 client.?.objecter ms_handle_connect 0x7f1b241457f0 2024-03-29T11:36:27.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 client.?.objecter ms_handle_connect 0x7f1b2413f650 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 client.?.objecter ms_handle_connect 0x7f1b240a9d70 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032881345 0 0) 0x7f1b240af1e0 con 0x7f1b241457f0 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b08004eb0 con 0x7f1b241457f0 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976927489 0 0) 0x7f1b240ad340 con 0x7f1b2413f650 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:27.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b240af1e0 con 0x7f1b2413f650 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682467827 0 0) 0x7f1b240ae340 con 0x7f1b240a9d70 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): my global_id is 4195 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b240ad340 con 0x7f1b240a9d70 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2947734054 0 0) 0x7f1b240af1e0 con 0x7f1b2413f650 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b240ae340 con 0x7f1b2413f650 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1199827408 0 0) 0x7f1b240ad340 con 0x7f1b240a9d70 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b240af1e0 con 0x7f1b240a9d70 2024-03-29T11:36:27.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1b14001970 con 0x7f1b2413f650 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:27.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 235594950 0 0) 0x7f1b08004eb0 con 0x7f1b241457f0 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b240ad340 con 0x7f1b241457f0 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1b20001b80 con 0x7f1b240a9d70 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:27.301 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1b1c001470 con 0x7f1b241457f0 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:27.302 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2498516476 0 0) 0x7f1b240ae340 con 0x7f1b2413f650 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 >> v1:172.21.15.191:6789/0 conn(0x7f1b240a9d70 legacy=0x7f1b241da600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 >> v1:172.21.15.100:6789/0 conn(0x7f1b241457f0 legacy=0x7f1b241e2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 monclient: found mon.c 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b240ad4d0 con 0x7f1b2413f650 2024-03-29T11:36:27.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b2b8d9640 5 monclient: authenticate success, global_id 4187 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b2b8d9640 10 monclient: _renew_subs 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b2b8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b241dad10 con 0x7f1b2413f650 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b14003610 con 0x7f1b2413f650 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1b140039a0 con 0x7f1b2413f650 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b19ffb640 10 monclient: dump: 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:27.304 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.285+0000 7f1b1b7fe640 4 set_mon_vals no callback set 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b2b8d9640 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b2b8d9640 10 monclient: _renew_subs 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b2b8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b24142060 con 0x7f1b2413f650 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b2b8d9640 1 librados: init done 2024-03-29T11:36:27.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1b14003f10 con 0x7f1b2413f650 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f1b140041c0 con 0x7f1b2413f650 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b19ffb640 10 client.4187.objecter ms_dispatch 0x7f1b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b19ffb640 3 client.4187.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b19ffb640 3 client.4187.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b19ffb640 20 client.4187.objecter dump_active .. 0 homeless 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b2b8d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b2b8d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b2b8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.286+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b24075f10 con 0x7f1b2413f650 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.288+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1b14004400 con 0x7f1b2413f650 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.288+0000 7f1b19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:27.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.288+0000 7f1b19ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:27.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.350+0000 7f1b2b8d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:27.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.350+0000 7f1b2b8d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:27.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.350+0000 7f1b2b8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:27.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.350+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/3855842613 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b240b3680 con 0x7f1b2413f650 2024-03-29T11:36:27.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.351+0000 7f1b19ffb640 1 -- 172.21.15.100:0/3855842613 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f1b140057c0 con 0x7f1b2413f650 2024-03-29T11:36:27.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.351+0000 7f1b19ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:27.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.351+0000 7f1b19ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:27.350 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:36:27.351 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.351+0000 7f1b2b8d9640 10 librados: watch_flush enter 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.351+0000 7f1b2b8d9640 10 librados: watch_flush exit 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.351+0000 7f1b2b8d9640 20 client.4187.objecter shutdown clearing up homeless session... 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.351+0000 7f1b2b8d9640 10 client.4187.objecter successfully canceled tick 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.352+0000 7f1b2b8d9640 10 monclient: shutdown 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.352+0000 7f1b2b8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.352+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/3855842613 >> v1:172.21.15.100:6790/0 conn(0x7f1b2413f650 legacy=0x7f1b24139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.352+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/3855842613 shutdown_connections 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.352+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/3855842613 >> 172.21.15.100:0/3855842613 conn(0x7f1b24066a00 msgr2=0x7f1b24066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.352+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/3855842613 shutdown_connections 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.352+0000 7f1b2b8d9640 1 -- 172.21.15.100:0/3855842613 wait complete. 2024-03-29T11:36:27.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:27.352+0000 7f1b2b8d9640 1 librados: shutdown 2024-03-29T11:36:27.360 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:27.718 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:27.719+0000 7f1b76fdb040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:36:27.740 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:27.741+0000 7fda32e32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:36:27.993 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:27.993+0000 7f1b76fdb040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:36:28.015 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:28.016+0000 7fda32e32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:36:28.043 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:28.044+0000 7f1b76fdb040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:36:28.066 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:28.067+0000 7fda32e32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:36:28.159 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:28.160+0000 7f1b76fdb040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:36:28.182 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-29T11:36:28.183+0000 7fda32e32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:36:29.172 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:29.173+0000 7f1b18437640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:36:29.173 INFO:tasks.ceph.mgr.y.smithi100.stderr:2024-03-29T11:36:29.174+0000 7f1b18437640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:36:30.229 INFO:tasks.ceph.osd.0.smithi100.stderr:2024-03-29T11:36:30.229+0000 7fb77ad14640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:36:30.265 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-29T11:36:30.266+0000 7f979fb73640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:36:30.272 INFO:tasks.ceph.osd.3.smithi100.stderr:2024-03-29T11:36:30.272+0000 7f84138d7640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:36:30.279 INFO:tasks.ceph.osd.2.smithi100.stderr:2024-03-29T11:36:30.279+0000 7fabb7e0d640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:36:30.313 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-29T11:36:30.314+0000 7f7ef0d24640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:36:30.322 INFO:tasks.ceph.osd.1.smithi100.stderr:2024-03-29T11:36:30.322+0000 7f6e4c068640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:36:30.329 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-29T11:36:30.331+0000 7fc289a10640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:36:30.383 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-29T11:36:30.384+0000 7f82c8781640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:36:34.167 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:34.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.324+0000 7fcff63fe640 10 monclient: get_monmap_and_config 2024-03-29T11:36:34.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.324+0000 7fcff63fe640 10 monclient: build_initial_monmap 2024-03-29T11:36:34.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.325+0000 7fcff63fe640 10 monclient: monmap: 2024-03-29T11:36:34.324 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:36:34.324 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:34.324 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:36:34.326008+0000 2024-03-29T11:36:34.324 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:36:34.326008+0000 2024-03-29T11:36:34.324 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:34.324 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:34.324 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:34.325 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:34.325 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:34.325 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:34.325 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.326+0000 7fcff63fe640 1 Processor -- start 2024-03-29T11:36:34.325 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.326+0000 7fcff63fe640 1 -- start start 2024-03-29T11:36:34.325 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.326+0000 7fcff63fe640 10 monclient: init 2024-03-29T11:36:34.326 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcff63fe640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:34.326 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcff63fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:34.326 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcff63fe640 10 monclient(hunting): picked mon.noname-c con 0x7fcff00b49f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:34.327 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcff63fe640 10 monclient(hunting): picked mon.noname-b con 0x7fcff00a9de0 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:34.327 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcff63fe640 10 monclient(hunting): picked mon.noname-a con 0x7fcff0142920 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:34.327 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcfef7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fcff00a9de0 0x7fcff0142210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:33446/0 (socket says 172.21.15.100:33446) 2024-03-29T11:36:34.327 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcfef7fe640 1 -- 172.21.15.100:0/3143880206 learned_addr learned my addr 172.21.15.100:0/3143880206 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:34.327 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcff63fe640 1 -- 172.21.15.100:0/3143880206 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff00ab120 con 0x7fcff0142920 2024-03-29T11:36:34.327 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcff63fe640 1 -- 172.21.15.100:0/3143880206 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff013fb10 con 0x7fcff00b49f0 2024-03-29T11:36:34.327 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcff63fe640 1 -- 172.21.15.100:0/3143880206 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff00b4420 con 0x7fcff00a9de0 2024-03-29T11:36:34.327 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcff63fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:34.327 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.327+0000 7fcff63fe640 10 monclient(hunting): authenticate will time out at 914.522919s 2024-03-29T11:36:34.328 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.328+0000 7fcfeeffd640 1 -- 172.21.15.100:0/3143880206 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316621438 0 0) 0x7fcff013fb10 con 0x7fcff00b49f0 2024-03-29T11:36:34.328 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.328+0000 7fcfeeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:34.328 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.328+0000 7fcfeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:34.328 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.328+0000 7fcfeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:34.328 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.328+0000 7fcfeeffd640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:36:34.328 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.328+0000 7fcfeeffd640 1 -- 172.21.15.100:0/3143880206 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd8003520 con 0x7fcff00b49f0 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 1 -- 172.21.15.100:0/3143880206 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2662672296 0 0) 0x7fcfd8003520 con 0x7fcff00b49f0 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 1 -- 172.21.15.100:0/3143880206 >> v1:172.21.15.100:6790/0 conn(0x7fcff00a9de0 legacy=0x7fcff0142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 1 -- 172.21.15.100:0/3143880206 >> v1:172.21.15.100:6789/0 conn(0x7fcff0142920 legacy=0x7fcff014d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 1 monclient: found mon.noname-c 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 1 -- 172.21.15.100:0/3143880206 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff0142db0 con 0x7fcff00b49f0 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcff63fe640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcff63fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:34.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 1 -- 172.21.15.100:0/3143880206 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fcfe0001690 con 0x7fcff00b49f0 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcfeeffd640 10 monclient: dump: 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:34.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.329+0000 7fcff63fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.330+0000 7fcfeeffd640 1 -- 172.21.15.100:0/3143880206 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfe00017f0 con 0x7fcff00b49f0 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.330+0000 7fcfeeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.330+0000 7fcff63fe640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.330+0000 7fcff63fe640 4 set_mon_vals no callback set 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.330+0000 7fcfeeffd640 1 -- 172.21.15.100:0/3143880206 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fcfe0001c00 con 0x7fcff00b49f0 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.330+0000 7fcfeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.330+0000 7fcfeeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.330+0000 7fcfeeffd640 10 monclient: dump: 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:34.332 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.331+0000 7fcff63fe640 10 monclient: shutdown 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.331+0000 7fcff63fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.331+0000 7fcff63fe640 1 -- 172.21.15.100:0/3143880206 >> v1:172.21.15.191:6789/0 conn(0x7fcff00b49f0 legacy=0x7fcff013bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.331+0000 7fcff63fe640 1 -- 172.21.15.100:0/3143880206 shutdown_connections 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.331+0000 7fcff63fe640 1 -- 172.21.15.100:0/3143880206 >> 172.21.15.100:0/3143880206 conn(0x7fcff0066a00 msgr2=0x7fcff0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.331+0000 7fcff63fe640 1 -- 172.21.15.100:0/3143880206 shutdown_connections 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.331+0000 7fcff63fe640 1 -- 172.21.15.100:0/3143880206 wait complete. 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.332+0000 7fcff63fe640 10 monclient: build_initial_monmap 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.332+0000 7fcff63fe640 10 monclient: monmap: 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:36:34.333 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.332+0000 7fcff63fe640 1 librados: starting msgr at 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.332+0000 7fcff63fe640 1 librados: starting objecter 2024-03-29T11:36:34.334 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.333+0000 7fcff63fe640 1 Processor -- start 2024-03-29T11:36:34.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.333+0000 7fcff63fe640 1 -- start start 2024-03-29T11:36:34.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.333+0000 7fcff63fe640 1 librados: setting wanted keys 2024-03-29T11:36:34.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.333+0000 7fcff63fe640 1 librados: calling monclient init 2024-03-29T11:36:34.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.333+0000 7fcff63fe640 10 monclient: init 2024-03-29T11:36:34.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcff63fe640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:34.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcff63fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:34.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcff63fe640 10 monclient(hunting): picked mon.noname-b con 0x7fcff0142920 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcff63fe640 10 monclient(hunting): picked mon.noname-a con 0x7fcff0145bd0 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcff63fe640 10 monclient(hunting): picked mon.noname-c con 0x7fcff013e860 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcff63fe640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff01df3d0 con 0x7fcff0145bd0 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcff63fe640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff01df860 con 0x7fcff0142920 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcff63fe640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff013d630 con 0x7fcff013e860 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcff63fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcff63fe640 10 monclient(hunting): authenticate will time out at 914.529356s 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcfeffff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fcff0142920 0x7fcff01deae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:57264/0 (socket says 172.21.15.100:57264) 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcfeffff640 1 -- 172.21.15.100:0/2392109784 learned_addr learned my addr 172.21.15.100:0/2392109784 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcfecff9640 10 client.?.objecter ms_handle_connect 0x7fcff0145bd0 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcfecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcfecff9640 10 client.?.objecter ms_handle_connect 0x7fcff0142920 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.334+0000 7fcfecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:34.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355130683 0 0) 0x7fcff01df860 con 0x7fcff0142920 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfdc004eb0 con 0x7fcff0142920 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430112182 0 0) 0x7fcff01df3d0 con 0x7fcff0145bd0 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff01df860 con 0x7fcff0145bd0 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 client.?.objecter ms_handle_connect 0x7fcff013e860 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:34.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1274528480 0 0) 0x7fcfdc004eb0 con 0x7fcff0142920 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcff01df3d0 con 0x7fcff0142920 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fcfe0001b40 con 0x7fcff0142920 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): dump: 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:34.340 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110071369 0 0) 0x7fcff013d630 con 0x7fcff013e860 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 10 monclient(hunting): my global_id is 4229 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.335+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfdc004eb0 con 0x7fcff013e860 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 36531832 0 0) 0x7fcff01df3d0 con 0x7fcff0142920 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 >> v1:172.21.15.100:6790/0 conn(0x7fcff013e860 legacy=0x7fcff013a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 >> v1:172.21.15.100:6789/0 conn(0x7fcff0145bd0 legacy=0x7fcff013e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 1 monclient: found mon.b 2024-03-29T11:36:34.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff013d7c0 con 0x7fcff0142920 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcff63fe640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcff63fe640 10 monclient: _renew_subs 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcff63fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcff63fe640 1 -- 172.21.15.100:0/2392109784 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff013ae10 con 0x7fcff0142920 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcff63fe640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcff63fe640 10 monclient: _renew_subs 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcff63fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:34.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcff63fe640 1 -- 172.21.15.100:0/2392109784 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff01e8010 con 0x7fcff0142920 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcff63fe640 1 librados: init done 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfe00038a0 con 0x7fcff0142920 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fcfe0001250 con 0x7fcff0142920 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfecff9640 10 monclient: dump: 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:34.343 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:34.344 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:34.344 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:34.344 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:34.344 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:34.344 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.336+0000 7fcfee7fc640 4 set_mon_vals no callback set 2024-03-29T11:36:34.344 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.337+0000 7fcff63fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:34.344 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.338+0000 7fcff63fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:34.344 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.338+0000 7fcff63fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:34.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.338+0000 7fcff63fe640 1 -- 172.21.15.100:0/2392109784 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd0005080 con 0x7fcff0142920 2024-03-29T11:36:34.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.339+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fcfe0003cf0 con 0x7fcff0142920 2024-03-29T11:36:34.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.341+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fcfe00849f0 con 0x7fcff0142920 2024-03-29T11:36:34.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.341+0000 7fcfecff9640 10 client.4243.objecter ms_dispatch 0x7fcff0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:34.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.341+0000 7fcfecff9640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:34.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.341+0000 7fcfecff9640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:34.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.341+0000 7fcfecff9640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-29T11:36:34.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.341+0000 7fcfef7fe640 10 monclient: get_auth_request con 0x7fcfdc06de00 auth_method 0 2024-03-29T11:36:34.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.341+0000 7fcfecff9640 10 client.4243.objecter ms_handle_connect 0x7fcfdc06de00 2024-03-29T11:36:34.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.347+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcfe0050410 con 0x7fcff0142920 2024-03-29T11:36:34.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.347+0000 7fcfecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:34.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.347+0000 7fcfecff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:34.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.712+0000 7fcff63fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:34.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.712+0000 7fcff63fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:34.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.712+0000 7fcff63fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:36:34.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.712+0000 7fcff63fe640 1 -- 172.21.15.100:0/2392109784 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfd0005330 con 0x7fcff0142920 2024-03-29T11:36:34.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.713+0000 7fcfecff9640 1 -- 172.21.15.100:0/2392109784 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fcfe0050410 con 0x7fcff0142920 2024-03-29T11:36:34.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.713+0000 7fcfecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:34.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.713+0000 7fcfecff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:34.713 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:36:34.713 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:34.716 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.717+0000 7fcff63fe640 10 librados: watch_flush enter 2024-03-29T11:36:34.716 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.717+0000 7fcff63fe640 10 librados: watch_flush exit 2024-03-29T11:36:34.716 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.717+0000 7fcff63fe640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-29T11:36:34.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.717+0000 7fcff63fe640 10 client.4243.objecter successfully canceled tick 2024-03-29T11:36:34.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.717+0000 7fcff63fe640 1 -- 172.21.15.100:0/2392109784 >> v1:172.21.15.100:6816/3220289657 conn(0x7fcfdc06de00 legacy=0x7fcfdc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:34.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.717+0000 7fcff63fe640 10 monclient: shutdown 2024-03-29T11:36:34.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.717+0000 7fcff63fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:34.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.717+0000 7fcff63fe640 1 -- 172.21.15.100:0/2392109784 >> v1:172.21.15.191:6789/0 conn(0x7fcff0142920 legacy=0x7fcff01deae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:34.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.718+0000 7fcff63fe640 1 -- 172.21.15.100:0/2392109784 shutdown_connections 2024-03-29T11:36:34.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.718+0000 7fcff63fe640 1 -- 172.21.15.100:0/2392109784 >> 172.21.15.100:0/2392109784 conn(0x7fcff0066a00 msgr2=0x7fcff01404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:34.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.718+0000 7fcff63fe640 1 -- 172.21.15.100:0/2392109784 shutdown_connections 2024-03-29T11:36:34.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.718+0000 7fcff63fe640 1 -- 172.21.15.100:0/2392109784 wait complete. 2024-03-29T11:36:34.718 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:34.718+0000 7fcff63fe640 1 librados: shutdown 2024-03-29T11:36:34.732 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:41.539 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:41.683 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.684+0000 7f0e83fed640 10 monclient: get_monmap_and_config 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.684+0000 7f0e83fed640 10 monclient: build_initial_monmap 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.684+0000 7f0e83fed640 10 monclient: monmap: 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:36:41.685660+0000 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:36:41.685660+0000 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:41.684 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:41.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.685+0000 7f0e83fed640 1 Processor -- start 2024-03-29T11:36:41.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.685+0000 7f0e83fed640 1 -- start start 2024-03-29T11:36:41.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.685+0000 7f0e83fed640 10 monclient: init 2024-03-29T11:36:41.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e83fed640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:41.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e83fed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:41.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e83fed640 10 monclient(hunting): picked mon.noname-a con 0x7f0e7c1457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:41.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e83fed640 10 monclient(hunting): picked mon.noname-c con 0x7f0e7c13f650 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:41.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e83fed640 10 monclient(hunting): picked mon.noname-b con 0x7f0e7c0a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:41.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e83fed640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7c0aaef0 con 0x7f0e7c1457f0 2024-03-29T11:36:41.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e83fed640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7c145bd0 con 0x7f0e7c13f650 2024-03-29T11:36:41.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e83fed640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7c145eb0 con 0x7f0e7c0a9d70 2024-03-29T11:36:41.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e83fed640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:41.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e83fed640 10 monclient(hunting): authenticate will time out at 921.881743s 2024-03-29T11:36:41.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e81588640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f0e7c13f650 0x7f0e7c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:46432/0 (socket says 172.21.15.100:46432) 2024-03-29T11:36:41.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.686+0000 7f0e81588640 1 -- 172.21.15.100:0/911353425 learned_addr learned my addr 172.21.15.100:0/911353425 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:41.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901580816 0 0) 0x7f0e7c145bd0 con 0x7f0e7c13f650 2024-03-29T11:36:41.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 10 monclient(hunting): _negotiate 2024-03-29T11:36:41.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:41.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:41.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:36:41.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e68003520 con 0x7f0e7c13f650 2024-03-29T11:36:41.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439160055 0 0) 0x7f0e7c145eb0 con 0x7f0e7c0a9d70 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 10 monclient(hunting): _negotiate 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.687+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7c145bd0 con 0x7f0e7c0a9d70 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2399330089 0 0) 0x7f0e68003520 con 0x7f0e7c13f650 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 >> v1:172.21.15.100:6790/0 conn(0x7f0e7c0a9d70 legacy=0x7f0e7c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 >> v1:172.21.15.100:6789/0 conn(0x7f0e7c1457f0 legacy=0x7f0e7c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 10 monclient: _finish_hunting 0 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 1 monclient: found mon.noname-c 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c146340 con 0x7f0e7c13f650 2024-03-29T11:36:41.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 10 monclient: _finish_auth 0 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 10 monclient: _check_auth_tickets 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e83fed640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e83fed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0e78001690 con 0x7f0e7c13f650 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e80d87640 10 monclient: dump: 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:41.689 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:41.690 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:41.690 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:41.690 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:41.690 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:41.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.688+0000 7f0e83fed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:41.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.690+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e78001870 con 0x7f0e7c13f650 2024-03-29T11:36:41.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.690+0000 7f0e80d87640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:41.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.690+0000 7f0e83fed640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:41.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.690+0000 7f0e83fed640 4 set_mon_vals no callback set 2024-03-29T11:36:41.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.690+0000 7f0e80d87640 1 -- 172.21.15.100:0/911353425 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0e78001690 con 0x7f0e7c13f650 2024-03-29T11:36:41.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.690+0000 7f0e80d87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:41.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.690+0000 7f0e80d87640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.690+0000 7f0e80d87640 10 monclient: dump: 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.691+0000 7f0e83fed640 10 monclient: shutdown 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.691+0000 7f0e83fed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:41.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.691+0000 7f0e83fed640 1 -- 172.21.15.100:0/911353425 >> v1:172.21.15.191:6789/0 conn(0x7f0e7c13f650 legacy=0x7f0e7c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.691+0000 7f0e83fed640 1 -- 172.21.15.100:0/911353425 shutdown_connections 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.691+0000 7f0e83fed640 1 -- 172.21.15.100:0/911353425 >> 172.21.15.100:0/911353425 conn(0x7f0e7c066a00 msgr2=0x7f0e7c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.691+0000 7f0e83fed640 1 -- 172.21.15.100:0/911353425 shutdown_connections 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.691+0000 7f0e83fed640 1 -- 172.21.15.100:0/911353425 wait complete. 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.692+0000 7f0e83fed640 10 monclient: build_initial_monmap 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.692+0000 7f0e83fed640 10 monclient: monmap: 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:41.693 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:41.694 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:41.694 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:41.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.692+0000 7f0e83fed640 1 librados: starting msgr at 2024-03-29T11:36:41.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.692+0000 7f0e83fed640 1 librados: starting objecter 2024-03-29T11:36:41.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.693+0000 7f0e83fed640 1 Processor -- start 2024-03-29T11:36:41.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.693+0000 7f0e83fed640 1 -- start start 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.693+0000 7f0e83fed640 1 librados: setting wanted keys 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.693+0000 7f0e83fed640 1 librados: calling monclient init 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.693+0000 7f0e83fed640 10 monclient: init 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.694+0000 7f0e83fed640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.694+0000 7f0e83fed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.694+0000 7f0e83fed640 10 monclient(hunting): picked mon.noname-b con 0x7f0e7c0a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.694+0000 7f0e83fed640 10 monclient(hunting): picked mon.noname-a con 0x7f0e7c13f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.694+0000 7f0e81588640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f0e7c13f650 0x7f0e7c137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:55792/0 (socket says 172.21.15.100:55792) 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.694+0000 7f0e81588640 1 -- 172.21.15.100:0/3907263625 learned_addr learned my addr 172.21.15.100:0/3907263625 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.694+0000 7f0e727fc640 10 client.?.objecter ms_handle_connect 0x7f0e7c13f650 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.694+0000 7f0e727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e83fed640 10 monclient(hunting): picked mon.noname-c con 0x7f0e7c1457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:41.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7c1cb9d0 con 0x7f0e7c13f650 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7c1cce40 con 0x7f0e7c0a9d70 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7c1cd120 con 0x7f0e7c1457f0 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e727fc640 10 client.?.objecter ms_handle_connect 0x7f0e7c1457f0 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525191323 0 0) 0x7f0e7c1cb9d0 con 0x7f0e7c13f650 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e83fed640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e83fed640 10 monclient(hunting): authenticate will time out at 921.890806s 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e727fc640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.695+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e60004eb0 con 0x7f0e7c13f650 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 client.?.objecter ms_handle_connect 0x7f0e7c0a9d70 2024-03-29T11:36:41.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184023506 0 0) 0x7f0e7c1cd120 con 0x7f0e7c1457f0 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7c1cb9d0 con 0x7f0e7c1457f0 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2749122378 0 0) 0x7f0e60004eb0 con 0x7f0e7c13f650 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e7c1cd120 con 0x7f0e7c13f650 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870650134 0 0) 0x7f0e7c1cce40 con 0x7f0e7c0a9d70 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:36:41.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e60004eb0 con 0x7f0e7c0a9d70 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0e78001690 con 0x7f0e7c13f650 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): dump: 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:41.705 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.696+0000 7f0e727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 651519463 0 0) 0x7f0e7c1cb9d0 con 0x7f0e7c1457f0 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e7c1cce40 con 0x7f0e7c1457f0 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0e6c0018b0 con 0x7f0e7c1457f0 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient(hunting): dump: 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:41.706 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1525951138 0 0) 0x7f0e7c1cd120 con 0x7f0e7c13f650 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 >> v1:172.21.15.100:6790/0 conn(0x7f0e7c1457f0 legacy=0x7f0e7c1e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 >> v1:172.21.15.191:6789/0 conn(0x7f0e7c0a9d70 legacy=0x7f0e7c1cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 1 monclient: found mon.a 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c1cbe60 con 0x7f0e7c13f650 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:41.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e83fed640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e83fed640 10 monclient: _renew_subs 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e83fed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7c1cdc50 con 0x7f0e7c13f650 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e83fed640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e83fed640 10 monclient: _renew_subs 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e83fed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7c1cb340 con 0x7f0e7c13f650 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e83fed640 1 librados: init done 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e78004b40 con 0x7f0e7c13f650 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f0e78004e50 con 0x7f0e7c13f650 2024-03-29T11:36:41.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.697+0000 7f0e727fc640 10 monclient: dump: 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.698+0000 7f0e73fff640 4 set_mon_vals no callback set 2024-03-29T11:36:41.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.699+0000 7f0e83fed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.699+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f0e7801ae20 con 0x7f0e7c13f650 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.699+0000 7f0e83fed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.699+0000 7f0e83fed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.699+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e64005080 con 0x7f0e7c13f650 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.700+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f0e78084cf0 con 0x7f0e7c13f650 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.700+0000 7f0e727fc640 10 client.4245.objecter ms_dispatch 0x7f0e7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.700+0000 7f0e727fc640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.700+0000 7f0e727fc640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.700+0000 7f0e727fc640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.700+0000 7f0e81d89640 10 monclient: get_auth_request con 0x7f0e6006df40 auth_method 0 2024-03-29T11:36:41.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.701+0000 7f0e727fc640 10 client.4245.objecter ms_handle_connect 0x7f0e6006df40 2024-03-29T11:36:41.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.707+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e780505d0 con 0x7f0e7c13f650 2024-03-29T11:36:41.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.707+0000 7f0e727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:41.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:41.707+0000 7f0e727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:42.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.058+0000 7f0e83fed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:42.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.058+0000 7f0e83fed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:42.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.058+0000 7f0e83fed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:42.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.058+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e64005330 con 0x7f0e7c13f650 2024-03-29T11:36:42.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.059+0000 7f0e727fc640 1 -- 172.21.15.100:0/3907263625 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f0e78052500 con 0x7f0e7c13f650 2024-03-29T11:36:42.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.059+0000 7f0e727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:42.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.059+0000 7f0e727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:42.059 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:36:42.059 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:42.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.062+0000 7f0e83fed640 10 librados: watch_flush enter 2024-03-29T11:36:42.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.062+0000 7f0e83fed640 10 librados: watch_flush exit 2024-03-29T11:36:42.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.062+0000 7f0e83fed640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T11:36:42.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.062+0000 7f0e83fed640 10 client.4245.objecter successfully canceled tick 2024-03-29T11:36:42.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.063+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 >> v1:172.21.15.100:6816/3220289657 conn(0x7f0e6006df40 legacy=0x7f0e60070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:42.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.063+0000 7f0e83fed640 10 monclient: shutdown 2024-03-29T11:36:42.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.063+0000 7f0e83fed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:42.063 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.063+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 >> v1:172.21.15.100:6789/0 conn(0x7f0e7c13f650 legacy=0x7f0e7c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:42.063 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.063+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 shutdown_connections 2024-03-29T11:36:42.063 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.063+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 >> 172.21.15.100:0/3907263625 conn(0x7f0e7c066a00 msgr2=0x7f0e7c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:42.063 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.063+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 shutdown_connections 2024-03-29T11:36:42.063 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.063+0000 7f0e83fed640 1 -- 172.21.15.100:0/3907263625 wait complete. 2024-03-29T11:36:42.063 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:42.063+0000 7f0e83fed640 1 librados: shutdown 2024-03-29T11:36:42.074 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:48.881 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:49.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.030+0000 7f1659251640 10 monclient: get_monmap_and_config 2024-03-29T11:36:49.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.030+0000 7f1659251640 10 monclient: build_initial_monmap 2024-03-29T11:36:49.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.030+0000 7f1659251640 10 monclient: monmap: 2024-03-29T11:36:49.030 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:36:49.030 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:49.031 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:36:49.031593+0000 2024-03-29T11:36:49.031 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:36:49.031593+0000 2024-03-29T11:36:49.031 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:49.031 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:49.031 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:49.031 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:49.031 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:49.031 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:49.032 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.032+0000 7f1659251640 1 Processor -- start 2024-03-29T11:36:49.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.032+0000 7f1659251640 1 -- start start 2024-03-29T11:36:49.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.032+0000 7f1659251640 10 monclient: init 2024-03-29T11:36:49.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.033+0000 7f1659251640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:49.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.033+0000 7f1659251640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:49.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.033+0000 7f1659251640 10 monclient(hunting): picked mon.noname-a con 0x7f1654071e60 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:49.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.033+0000 7f1659251640 10 monclient(hunting): picked mon.noname-c con 0x7f16540a9c70 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:49.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.033+0000 7f1659251640 10 monclient(hunting): picked mon.noname-b con 0x7f165413f8a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:49.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.033+0000 7f1659251640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16540aaeb0 con 0x7f1654071e60 2024-03-29T11:36:49.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.033+0000 7f1659251640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1654072c50 con 0x7f16540a9c70 2024-03-29T11:36:49.034 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.033+0000 7f1659251640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1654145ac0 con 0x7f165413f8a0 2024-03-29T11:36:49.034 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.033+0000 7f1659251640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:49.034 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.033+0000 7f1659251640 10 monclient(hunting): authenticate will time out at 929.228815s 2024-03-29T11:36:49.035 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.034+0000 7f16537fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f16540a9c70 0x7f16541453b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:58972/0 (socket says 172.21.15.100:58972) 2024-03-29T11:36:49.035 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.034+0000 7f16537fe640 1 -- 172.21.15.100:0/526374706 learned_addr learned my addr 172.21.15.100:0/526374706 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:49.035 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.035+0000 7f16537fe640 1 -- 172.21.15.100:0/526374706 >> v1:172.21.15.191:6789/0 conn(0x7f16540a9c70 legacy=0x7f16541453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:36:49.035 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.035+0000 7f16537fe640 1 -- 172.21.15.100:0/526374706 >> v1:172.21.15.191:6789/0 conn(0x7f16540a9c70 legacy=0x7f16541453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:36:49.035 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.035+0000 7f16537fe640 1 --1- 172.21.15.100:0/526374706 >> v1:172.21.15.191:6789/0 conn(0x7f16540a9c70 0x7f16541453b0 :-1 s=OPENED pgs=66631 cs=1 l=1).handle_message read tag failed 2024-03-29T11:36:49.035 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.035+0000 7f16537fe640 1 --1- 172.21.15.100:0/526374706 >> v1:172.21.15.191:6789/0 conn(0x7f16540a9c70 0x7f16541453b0 :-1 s=OPENED pgs=66631 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:36:49.035 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.035+0000 7f1652ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.191:6789/0 2024-03-29T11:36:49.040 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.037+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634798977 0 0) 0x7f1654145ac0 con 0x7f165413f8a0 2024-03-29T11:36:49.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.037+0000 7f1652ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:49.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.037+0000 7f1652ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:49.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.037+0000 7f1652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:49.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:36:49.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f163c003520 con 0x7f165413f8a0 2024-03-29T11:36:49.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582491525 0 0) 0x7f16540aaeb0 con 0x7f1654071e60 2024-03-29T11:36:49.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:49.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:49.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:49.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:36:49.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1654145ac0 con 0x7f1654071e60 2024-03-29T11:36:49.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1546835201 0 0) 0x7f1654145ac0 con 0x7f1654071e60 2024-03-29T11:36:49.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 >> v1:172.21.15.100:6790/0 conn(0x7f165413f8a0 legacy=0x7f165414d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:49.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 >> v1:172.21.15.191:6789/0 conn(0x7f16540a9c70 legacy=0x7f16541453b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:36:49.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:49.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 monclient: found mon.noname-a 2024-03-29T11:36:49.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:49.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:36:49.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1654145c50 con 0x7f1654071e60 2024-03-29T11:36:49.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:49.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:49.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:49.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1644001560 con 0x7f1654071e60 2024-03-29T11:36:49.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:49.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:49.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: dump: 2024-03-29T11:36:49.043 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1659251640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1659251640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1644001780 con 0x7f1654071e60 2024-03-29T11:36:49.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1659251640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1659251640 4 set_mon_vals no callback set 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 1 -- 172.21.15.100:0/526374706 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1644001b10 con 0x7f1654071e60 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.038+0000 7f1652ffd640 10 monclient: dump: 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:49.045 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.039+0000 7f1659251640 10 monclient: shutdown 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.039+0000 7f1659251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.039+0000 7f1659251640 1 -- 172.21.15.100:0/526374706 >> v1:172.21.15.100:6789/0 conn(0x7f1654071e60 legacy=0x7f165413f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.039+0000 7f1659251640 1 -- 172.21.15.100:0/526374706 shutdown_connections 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.039+0000 7f1659251640 1 -- 172.21.15.100:0/526374706 >> 172.21.15.100:0/526374706 conn(0x7f16540b3a30 msgr2=0x7f16540b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.040+0000 7f1659251640 1 -- 172.21.15.100:0/526374706 shutdown_connections 2024-03-29T11:36:49.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.040+0000 7f1659251640 1 -- 172.21.15.100:0/526374706 wait complete. 2024-03-29T11:36:49.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.040+0000 7f1659251640 10 monclient: build_initial_monmap 2024-03-29T11:36:49.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.040+0000 7f1659251640 10 monclient: monmap: 2024-03-29T11:36:49.048 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:36:49.048 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:49.048 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:36:49.048 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:36:49.048 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:49.048 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:49.048 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:49.048 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:49.048 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:49.049 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:49.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.041+0000 7f1659251640 1 librados: starting msgr at 2024-03-29T11:36:49.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.041+0000 7f1659251640 1 librados: starting objecter 2024-03-29T11:36:49.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.041+0000 7f1659251640 1 Processor -- start 2024-03-29T11:36:49.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.041+0000 7f1659251640 1 -- start start 2024-03-29T11:36:49.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.041+0000 7f1659251640 1 librados: setting wanted keys 2024-03-29T11:36:49.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.041+0000 7f1659251640 1 librados: calling monclient init 2024-03-29T11:36:49.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.041+0000 7f1659251640 10 monclient: init 2024-03-29T11:36:49.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.042+0000 7f1659251640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:49.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.042+0000 7f1659251640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.042+0000 7f1659251640 10 monclient(hunting): picked mon.noname-c con 0x7f1654071e60 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.043+0000 7f1659251640 10 monclient(hunting): picked mon.noname-b con 0x7f16540a9c70 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.043+0000 7f16537fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f16540a9c70 0x7f1654138ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:59106/0 (socket says 172.21.15.100:59106) 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.045+0000 7f16537fe640 1 -- 172.21.15.100:0/3841448731 learned_addr learned my addr 172.21.15.100:0/3841448731 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.045+0000 7f1659251640 10 monclient(hunting): picked mon.noname-a con 0x7f165413f8a0 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.045+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165406ea60 con 0x7f165413f8a0 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.045+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1654139e10 con 0x7f16540a9c70 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.045+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165406ecb0 con 0x7f1654071e60 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.045+0000 7f1659251640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.045+0000 7f1659251640 10 monclient(hunting): authenticate will time out at 929.240626s 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.046+0000 7f1650ff9640 10 client.?.objecter ms_handle_connect 0x7f1654071e60 2024-03-29T11:36:49.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.046+0000 7f1650ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:49.051 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.047+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408605161 0 0) 0x7f165406ecb0 con 0x7f1654071e60 2024-03-29T11:36:49.051 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.047+0000 7f1650ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:49.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.049+0000 7f1650ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:49.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.049+0000 7f1650ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:49.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.049+0000 7f1650ff9640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.049+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1640004eb0 con 0x7f1654071e60 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.049+0000 7f1650ff9640 10 client.?.objecter ms_handle_connect 0x7f16540a9c70 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.049+0000 7f1650ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.049+0000 7f1650ff9640 10 client.?.objecter ms_handle_connect 0x7f165413f8a0 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.049+0000 7f1650ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1581030374 0 0) 0x7f1640004eb0 con 0x7f1654071e60 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f165406ecb0 con 0x7f1654071e60 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f16440014a0 con 0x7f1654071e60 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:49.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:49.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780706959 0 0) 0x7f165406ea60 con 0x7f165413f8a0 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1640004eb0 con 0x7f165413f8a0 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3776887272 0 0) 0x7f165406ecb0 con 0x7f1654071e60 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 >> v1:172.21.15.191:6789/0 conn(0x7f16540a9c70 legacy=0x7f1654138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 >> v1:172.21.15.100:6789/0 conn(0x7f165413f8a0 legacy=0x7f1654139700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 1 monclient: found mon.c 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:49.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165406ef90 con 0x7f1654071e60 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1650ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1659251640 5 monclient: authenticate success, global_id 4241 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1659251640 10 monclient: _renew_subs 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1659251640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165406f270 con 0x7f1654071e60 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1659251640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1659251640 10 monclient: _renew_subs 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1659251640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:49.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16540700f0 con 0x7f1654071e60 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.050+0000 7f1659251640 1 librados: init done 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.053+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1644003910 con 0x7f1654071e60 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.053+0000 7f1650ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.053+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1644003d60 con 0x7f1654071e60 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.053+0000 7f1650ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.053+0000 7f1650ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.053+0000 7f1650ff9640 10 monclient: dump: 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:49.059 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.053+0000 7f1659251640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.053+0000 7f1659251640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.053+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f1644003f80 con 0x7f1654071e60 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.053+0000 7f1659251640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.054+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165404f730 con 0x7f1654071e60 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.054+0000 7f1651ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:49.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.054+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f16440838f0 con 0x7f1654071e60 2024-03-29T11:36:49.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.054+0000 7f1650ff9640 10 client.4241.objecter ms_dispatch 0x7f1654000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:49.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.054+0000 7f1650ff9640 3 client.4241.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:49.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.054+0000 7f1650ff9640 3 client.4241.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:49.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.054+0000 7f16537fe640 10 monclient: get_auth_request con 0x7f164006dde0 auth_method 0 2024-03-29T11:36:49.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.054+0000 7f1650ff9640 20 client.4241.objecter dump_active .. 0 homeless 2024-03-29T11:36:49.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.055+0000 7f1650ff9640 10 client.4241.objecter ms_handle_connect 0x7f164006dde0 2024-03-29T11:36:49.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.062+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f164404f1d0 con 0x7f1654071e60 2024-03-29T11:36:49.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.062+0000 7f1650ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:49.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.062+0000 7f1650ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:49.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.383+0000 7f1659251640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:49.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.383+0000 7f1659251640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:49.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.383+0000 7f1659251640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:36:49.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.383+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1654074d80 con 0x7f1654071e60 2024-03-29T11:36:49.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.384+0000 7f1650ff9640 1 -- 172.21.15.100:0/3841448731 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f1644051100 con 0x7f1654071e60 2024-03-29T11:36:49.384 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.384+0000 7f1650ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:49.384 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.384+0000 7f1650ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:49.385 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:36:49.385 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:49.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.388+0000 7f1659251640 10 librados: watch_flush enter 2024-03-29T11:36:49.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.390+0000 7f1659251640 10 librados: watch_flush exit 2024-03-29T11:36:49.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.390+0000 7f1659251640 20 client.4241.objecter shutdown clearing up homeless session... 2024-03-29T11:36:49.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.390+0000 7f1659251640 10 client.4241.objecter successfully canceled tick 2024-03-29T11:36:49.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.390+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 >> v1:172.21.15.100:6816/3220289657 conn(0x7f164006dde0 legacy=0x7f16400702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:49.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.390+0000 7f1659251640 10 monclient: shutdown 2024-03-29T11:36:49.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.390+0000 7f1659251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:49.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.390+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 >> v1:172.21.15.100:6790/0 conn(0x7f1654071e60 legacy=0x7f165406f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:49.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.390+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 shutdown_connections 2024-03-29T11:36:49.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.390+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 >> 172.21.15.100:0/3841448731 conn(0x7f16540b3a30 msgr2=0x7f16540ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:49.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.391+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 shutdown_connections 2024-03-29T11:36:49.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.391+0000 7f1659251640 1 -- 172.21.15.100:0/3841448731 wait complete. 2024-03-29T11:36:49.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:49.391+0000 7f1659251640 1 librados: shutdown 2024-03-29T11:36:49.407 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:56.213 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:56.365 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.366+0000 7fc8cd0b3640 10 monclient: get_monmap_and_config 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.366+0000 7fc8cd0b3640 10 monclient: build_initial_monmap 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.366+0000 7fc8cd0b3640 10 monclient: monmap: 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:36:56.367470+0000 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:36:56.367470+0000 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:36:56.366 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:56.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.367+0000 7fc8cd0b3640 1 Processor -- start 2024-03-29T11:36:56.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.367+0000 7fc8cd0b3640 1 -- start start 2024-03-29T11:36:56.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.367+0000 7fc8cd0b3640 10 monclient: init 2024-03-29T11:36:56.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.368+0000 7fc8cd0b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:56.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.368+0000 7fc8cd0b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:56.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.368+0000 7fc8cd0b3640 10 monclient(hunting): picked mon.noname-a con 0x7fc8c80b2800 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:56.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.368+0000 7fc8cd0b3640 10 monclient(hunting): picked mon.noname-b con 0x7fc8c8142b20 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:56.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.368+0000 7fc8cd0b3640 10 monclient(hunting): picked mon.noname-c con 0x7fc8c813c040 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:56.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.368+0000 7fc8cd0b3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c80aad10 con 0x7fc8c80b2800 2024-03-29T11:36:56.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.368+0000 7fc8cd0b3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c8141f30 con 0x7fc8c813c040 2024-03-29T11:36:56.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.368+0000 7fc8cd0b3640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c80aaf60 con 0x7fc8c8142b20 2024-03-29T11:36:56.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.368+0000 7fc8cd0b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:56.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.368+0000 7fc8cd0b3640 10 monclient(hunting): authenticate will time out at 936.563645s 2024-03-29T11:36:56.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.371+0000 7fc8cc8b2640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fc8c813c040 0x7fc8c814d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:44952/0 (socket says 172.21.15.100:44952) 2024-03-29T11:36:56.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.371+0000 7fc8cc8b2640 1 -- 172.21.15.100:0/4090873236 learned_addr learned my addr 172.21.15.100:0/4090873236 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:56.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.371+0000 7fc8c6ffd640 1 -- 172.21.15.100:0/4090873236 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010534458 0 0) 0x7fc8c8141f30 con 0x7fc8c813c040 2024-03-29T11:36:56.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.371+0000 7fc8c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:56.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.371+0000 7fc8c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:56.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.371+0000 7fc8c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:56.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.371+0000 7fc8c6ffd640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:36:56.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.371+0000 7fc8c6ffd640 1 -- 172.21.15.100:0/4090873236 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b0003520 con 0x7fc8c813c040 2024-03-29T11:36:56.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 1 -- 172.21.15.100:0/4090873236 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1841946264 0 0) 0x7fc8b0003520 con 0x7fc8c813c040 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 1 -- 172.21.15.100:0/4090873236 >> v1:172.21.15.100:6790/0 conn(0x7fc8c8142b20 legacy=0x7fc8c813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 1 -- 172.21.15.100:0/4090873236 >> v1:172.21.15.100:6789/0 conn(0x7fc8c80b2800 legacy=0x7fc8c8142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 1 -- 172.21.15.100:0/4090873236 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c80b3900 con 0x7fc8c813c040 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 1 -- 172.21.15.100:0/4090873236 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc8c0001a10 con 0x7fc8c813c040 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:56.372 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: dump: 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8cd0b3640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8cd0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:56.373 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 1 -- 172.21.15.100:0/4090873236 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8c0001c30 con 0x7fc8c813c040 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 1 -- 172.21.15.100:0/4090873236 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc8c0001fc0 con 0x7fc8c813c040 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8cd0b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8cd0b3640 4 set_mon_vals no callback set 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.372+0000 7fc8c6ffd640 10 monclient: dump: 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:56.374 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:56.375 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:56.375 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:56.375 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:56.375 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.373+0000 7fc8cd0b3640 10 monclient: shutdown 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.373+0000 7fc8cd0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.373+0000 7fc8cd0b3640 1 -- 172.21.15.100:0/4090873236 >> v1:172.21.15.191:6789/0 conn(0x7fc8c813c040 legacy=0x7fc8c814d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.373+0000 7fc8cd0b3640 1 -- 172.21.15.100:0/4090873236 shutdown_connections 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.373+0000 7fc8cd0b3640 1 -- 172.21.15.100:0/4090873236 >> 172.21.15.100:0/4090873236 conn(0x7fc8c806d0e0 msgr2=0x7fc8c8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.373+0000 7fc8cd0b3640 1 -- 172.21.15.100:0/4090873236 shutdown_connections 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.373+0000 7fc8cd0b3640 1 -- 172.21.15.100:0/4090873236 wait complete. 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.374+0000 7fc8cd0b3640 10 monclient: build_initial_monmap 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.374+0000 7fc8cd0b3640 10 monclient: monmap: 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:56.376 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.375+0000 7fc8cd0b3640 1 librados: starting msgr at 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.375+0000 7fc8cd0b3640 1 librados: starting objecter 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.375+0000 7fc8cd0b3640 1 Processor -- start 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.375+0000 7fc8cd0b3640 1 -- start start 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.375+0000 7fc8cd0b3640 1 librados: setting wanted keys 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.375+0000 7fc8cd0b3640 1 librados: calling monclient init 2024-03-29T11:36:56.377 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.375+0000 7fc8cd0b3640 10 monclient: init 2024-03-29T11:36:56.378 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.376+0000 7fc8cd0b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.376+0000 7fc8cd0b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.376+0000 7fc8cd0b3640 10 monclient(hunting): picked mon.noname-c con 0x7fc8c80b2800 addr v1:172.21.15.100:6790/0 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.376+0000 7fc8cd0b3640 10 monclient(hunting): picked mon.noname-b con 0x7fc8c813c040 addr v1:172.21.15.191:6789/0 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.376+0000 7fc8cd0b3640 10 monclient(hunting): picked mon.noname-a con 0x7fc8c8142b20 addr v1:172.21.15.100:6789/0 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.376+0000 7fc8cd0b3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c806e560 con 0x7fc8c8142b20 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.376+0000 7fc8cd0b3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c81454e0 con 0x7fc8c813c040 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.376+0000 7fc8cd0b3640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c8144fe0 con 0x7fc8c80b2800 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.376+0000 7fc8cd0b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.376+0000 7fc8cd0b3640 10 monclient(hunting): authenticate will time out at 936.571643s 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.377+0000 7fc8c77fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fc8c813c040 0x7fc8c81e6c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:45232/0 (socket says 172.21.15.100:45232) 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.377+0000 7fc8c77fe640 1 -- 172.21.15.100:0/2700634519 learned_addr learned my addr 172.21.15.100:0/2700634519 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.378+0000 7fc8c4ff9640 10 client.?.objecter ms_handle_connect 0x7fc8c813c040 2024-03-29T11:36:56.379 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.378+0000 7fc8c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:56.380 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.378+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808163913 0 0) 0x7fc8c81454e0 con 0x7fc8c813c040 2024-03-29T11:36:56.380 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.378+0000 7fc8c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:56.380 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.378+0000 7fc8c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:56.380 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.378+0000 7fc8c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:56.380 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.378+0000 7fc8c4ff9640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:36:56.380 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.378+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b4004eb0 con 0x7fc8c813c040 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 client.?.objecter ms_handle_connect 0x7fc8c80b2800 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 client.?.objecter ms_handle_connect 0x7fc8c8142b20 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893534951 0 0) 0x7fc8c8144fe0 con 0x7fc8c80b2800 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c81454e0 con 0x7fc8c80b2800 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137605841 0 0) 0x7fc8c806e560 con 0x7fc8c8142b20 2024-03-29T11:36:56.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:56.382 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:56.382 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:56.382 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.380+0000 7fc8c4ff9640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:36:56.382 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.381+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c8144fe0 con 0x7fc8c8142b20 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1580322024 0 0) 0x7fc8c81454e0 con 0x7fc8c80b2800 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8c806e560 con 0x7fc8c80b2800 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3963833126 0 0) 0x7fc8b4004eb0 con 0x7fc8c813c040 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8c81454e0 con 0x7fc8c813c040 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112728476 0 0) 0x7fc8c8144fe0 con 0x7fc8c8142b20 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8b4004eb0 con 0x7fc8c8142b20 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc8b80017a0 con 0x7fc8c80b2800 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:56.383 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc8bc0018c0 con 0x7fc8c813c040 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:56.384 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc8c0001da0 con 0x7fc8c8142b20 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:56.385 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.382+0000 7fc8c4ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1146212126 0 0) 0x7fc8b4004eb0 con 0x7fc8c8142b20 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 >> v1:172.21.15.100:6790/0 conn(0x7fc8c80b2800 legacy=0x7fc8c81e0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 >> v1:172.21.15.191:6789/0 conn(0x7fc8c813c040 legacy=0x7fc8c81e6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:56.386 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:56.387 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 1 monclient: found mon.a 2024-03-29T11:36:56.387 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:56.387 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:56.387 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c8145170 con 0x7fc8c8142b20 2024-03-29T11:36:56.387 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:56.387 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:56.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:56.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8cd0b3640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:36:56.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8cd0b3640 10 monclient: _renew_subs 2024-03-29T11:36:56.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8cd0b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:56.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8cd0b3640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c81e10a0 con 0x7fc8c8142b20 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8cd0b3640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8cd0b3640 10 monclient: _renew_subs 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8cd0b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8cd0b3640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c8140e90 con 0x7fc8c8142b20 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.383+0000 7fc8cd0b3640 1 librados: init done 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.384+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8c0001f00 con 0x7fc8c8142b20 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.384+0000 7fc8c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.384+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc8c00020f0 con 0x7fc8c8142b20 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.384+0000 7fc8c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.384+0000 7fc8c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.384+0000 7fc8c4ff9640 10 monclient: dump: 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:36:56.389 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.385+0000 7fc8ae7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.385+0000 7fc8ae7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.385+0000 7fc8ae7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.385+0000 7fc8ae7fc640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a8005080 con 0x7fc8c8142b20 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.386+0000 7fc8c67fc640 4 set_mon_vals no callback set 2024-03-29T11:36:56.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.386+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fc8c001af80 con 0x7fc8c8142b20 2024-03-29T11:36:56.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.386+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fc8c0084c10 con 0x7fc8c8142b20 2024-03-29T11:36:56.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.386+0000 7fc8c4ff9640 10 client.4257.objecter ms_dispatch 0x7fc8c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:56.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.386+0000 7fc8c4ff9640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:56.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.386+0000 7fc8c4ff9640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:56.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.386+0000 7fc8c7fff640 10 monclient: get_auth_request con 0x7fc8b406e280 auth_method 0 2024-03-29T11:36:56.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.386+0000 7fc8c4ff9640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T11:36:56.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.387+0000 7fc8c4ff9640 10 client.4257.objecter ms_handle_connect 0x7fc8b406e280 2024-03-29T11:36:56.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.396+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8c0050460 con 0x7fc8c8142b20 2024-03-29T11:36:56.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.396+0000 7fc8c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:56.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.396+0000 7fc8c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:56.736 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.736+0000 7fc8ae7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:56.736 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.736+0000 7fc8ae7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:56.736 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.736+0000 7fc8ae7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:36:56.736 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.736+0000 7fc8ae7fc640 1 -- 172.21.15.100:0/2700634519 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8a8005330 con 0x7fc8c8142b20 2024-03-29T11:36:56.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.737+0000 7fc8c4ff9640 1 -- 172.21.15.100:0/2700634519 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fc8c0052560 con 0x7fc8c8142b20 2024-03-29T11:36:56.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.737+0000 7fc8c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:56.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.737+0000 7fc8c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:56.737 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:36:56.737 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:56.742 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.741+0000 7fc8ae7fc640 10 librados: watch_flush enter 2024-03-29T11:36:56.742 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.741+0000 7fc8ae7fc640 10 librados: watch_flush exit 2024-03-29T11:36:56.742 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.741+0000 7fc8ae7fc640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T11:36:56.742 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.741+0000 7fc8ae7fc640 10 client.4257.objecter successfully canceled tick 2024-03-29T11:36:56.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.743+0000 7fc8ae7fc640 1 -- 172.21.15.100:0/2700634519 >> v1:172.21.15.100:6816/3220289657 conn(0x7fc8b406e280 legacy=0x7fc8b4070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:56.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.743+0000 7fc8ae7fc640 10 monclient: shutdown 2024-03-29T11:36:56.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.743+0000 7fc8ae7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:56.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.743+0000 7fc8ae7fc640 1 -- 172.21.15.100:0/2700634519 >> v1:172.21.15.100:6789/0 conn(0x7fc8c8142b20 legacy=0x7fc8c81ef010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:56.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.743+0000 7fc8ae7fc640 1 -- 172.21.15.100:0/2700634519 shutdown_connections 2024-03-29T11:36:56.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.743+0000 7fc8ae7fc640 1 -- 172.21.15.100:0/2700634519 >> 172.21.15.100:0/2700634519 conn(0x7fc8c806d0e0 msgr2=0x7fc8c813b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:56.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.743+0000 7fc8ae7fc640 1 -- 172.21.15.100:0/2700634519 shutdown_connections 2024-03-29T11:36:56.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.747+0000 7fc8ae7fc640 1 -- 172.21.15.100:0/2700634519 wait complete. 2024-03-29T11:36:56.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:36:56.747+0000 7fc8ae7fc640 1 librados: shutdown 2024-03-29T11:36:56.764 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:03.570 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:03.721 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.719+0000 7f6cbdf80640 10 monclient: get_monmap_and_config 2024-03-29T11:37:03.721 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.719+0000 7f6cbdf80640 10 monclient: build_initial_monmap 2024-03-29T11:37:03.721 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.719+0000 7f6cbdf80640 10 monclient: monmap: 2024-03-29T11:37:03.721 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:03.721 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:03.722 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:37:03.720192+0000 2024-03-29T11:37:03.722 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:37:03.720192+0000 2024-03-29T11:37:03.722 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:03.722 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:03.722 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:03.722 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:37:03.722 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:37:03.722 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:03.726 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.720+0000 7f6cbdf80640 1 Processor -- start 2024-03-29T11:37:03.726 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.720+0000 7f6cbdf80640 1 -- start start 2024-03-29T11:37:03.726 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.720+0000 7f6cbdf80640 10 monclient: init 2024-03-29T11:37:03.726 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.721+0000 7f6cbdf80640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:03.726 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.721+0000 7f6cbdf80640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:03.726 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.721+0000 7f6cbdf80640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb80b2800 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:03.726 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.721+0000 7f6cbdf80640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb80a9c00 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:03.727 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.723+0000 7f6cbdf80640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb80b38c0 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:03.727 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.723+0000 7f6cbdf80640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb8141eb0 con 0x7f6cb80b2800 2024-03-29T11:37:03.727 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.723+0000 7f6cbdf80640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb80b22d0 con 0x7f6cb80b38c0 2024-03-29T11:37:03.727 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.723+0000 7f6cbdf80640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb8142d30 con 0x7f6cb80a9c00 2024-03-29T11:37:03.727 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.723+0000 7f6cbdf80640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:03.727 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.723+0000 7f6cbdf80640 10 monclient(hunting): authenticate will time out at 943.918804s 2024-03-29T11:37:03.730 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.729+0000 7f6cb7fff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f6cb80a9c00 0x7f6cb813bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:53107/0 (socket says 172.21.15.100:53107) 2024-03-29T11:37:03.730 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.729+0000 7f6cb7fff640 1 -- 172.21.15.100:0/2726119716 learned_addr learned my addr 172.21.15.100:0/2726119716 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:03.730 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.729+0000 7f6cb77fe640 1 -- 172.21.15.100:0/2726119716 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147334084 0 0) 0x7f6cb8142d30 con 0x7f6cb80a9c00 2024-03-29T11:37:03.730 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.729+0000 7f6cb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:03.730 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.729+0000 7f6cb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:03.730 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.729+0000 7f6cb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:03.730 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.729+0000 7f6cb77fe640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:37:03.730 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.729+0000 7f6cb77fe640 1 -- 172.21.15.100:0/2726119716 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca8003520 con 0x7f6cb80a9c00 2024-03-29T11:37:03.730 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 1 -- 172.21.15.100:0/2726119716 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995641256 0 0) 0x7f6ca8003520 con 0x7f6cb80a9c00 2024-03-29T11:37:03.730 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 1 -- 172.21.15.100:0/2726119716 >> v1:172.21.15.191:6789/0 conn(0x7f6cb80b38c0 legacy=0x7f6cb814d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 1 -- 172.21.15.100:0/2726119716 >> v1:172.21.15.100:6789/0 conn(0x7f6cb80b2800 legacy=0x7f6cb8142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 1 monclient: found mon.noname-b 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 1 -- 172.21.15.100:0/2726119716 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb80aaf90 con 0x7f6cb80a9c00 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 1 -- 172.21.15.100:0/2726119716 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6ca4001690 con 0x7f6cb80a9c00 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: dump: 2024-03-29T11:37:03.731 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cbdf80640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cbdf80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 1 -- 172.21.15.100:0/2726119716 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ca40018f0 con 0x7f6cb80a9c00 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:03.732 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 1 -- 172.21.15.100:0/2726119716 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6ca4001cc0 con 0x7f6cb80a9c00 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cb77fe640 10 monclient: dump: 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cbdf80640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:03.733 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.730+0000 7f6cbdf80640 4 set_mon_vals no callback set 2024-03-29T11:37:03.734 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.731+0000 7f6cbdf80640 10 monclient: shutdown 2024-03-29T11:37:03.734 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.731+0000 7f6cbdf80640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:03.734 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.731+0000 7f6cbdf80640 1 -- 172.21.15.100:0/2726119716 >> v1:172.21.15.100:6790/0 conn(0x7f6cb80a9c00 legacy=0x7f6cb813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:03.734 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.731+0000 7f6cbdf80640 1 -- 172.21.15.100:0/2726119716 shutdown_connections 2024-03-29T11:37:03.734 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.731+0000 7f6cbdf80640 1 -- 172.21.15.100:0/2726119716 >> 172.21.15.100:0/2726119716 conn(0x7f6cb8066a00 msgr2=0x7f6cb8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.732+0000 7f6cbdf80640 1 -- 172.21.15.100:0/2726119716 shutdown_connections 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.732+0000 7f6cbdf80640 1 -- 172.21.15.100:0/2726119716 wait complete. 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.733+0000 7f6cbdf80640 10 monclient: build_initial_monmap 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.733+0000 7f6cbdf80640 10 monclient: monmap: 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:37:03.735 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:03.736 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.733+0000 7f6cbdf80640 1 librados: starting msgr at 2024-03-29T11:37:03.736 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.734+0000 7f6cbdf80640 1 librados: starting objecter 2024-03-29T11:37:03.736 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.734+0000 7f6cbdf80640 1 Processor -- start 2024-03-29T11:37:03.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.734+0000 7f6cbdf80640 1 -- start start 2024-03-29T11:37:03.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.734+0000 7f6cbdf80640 1 librados: setting wanted keys 2024-03-29T11:37:03.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.734+0000 7f6cbdf80640 1 librados: calling monclient init 2024-03-29T11:37:03.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.734+0000 7f6cbdf80640 10 monclient: init 2024-03-29T11:37:03.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.736+0000 7f6cbdf80640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:03.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.736+0000 7f6cbdf80640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:03.738 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.736+0000 7f6cbdf80640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb80a9c00 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:03.738 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.736+0000 7f6cbdf80640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb80b2800 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:03.738 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.737+0000 7f6cbdf80640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb80b38c0 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:03.738 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.737+0000 7f6cbdf80640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb80ad7d0 con 0x7f6cb80b2800 2024-03-29T11:37:03.738 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.737+0000 7f6cbdf80640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb80aeff0 con 0x7f6cb80a9c00 2024-03-29T11:37:03.738 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.737+0000 7f6cbdf80640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb80ada20 con 0x7f6cb80b38c0 2024-03-29T11:37:03.738 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.737+0000 7f6cbdf80640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:03.738 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.737+0000 7f6cbdf80640 10 monclient(hunting): authenticate will time out at 943.933027s 2024-03-29T11:37:03.742 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.743+0000 7f6cbcf7e640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f6cb80a9c00 0x7f6cb80ad0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:40351/0 (socket says 172.21.15.100:40351) 2024-03-29T11:37:03.742 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.743+0000 7f6cbcf7e640 1 -- 172.21.15.100:0/751330700 learned_addr learned my addr 172.21.15.100:0/751330700 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:03.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.743+0000 7f6cb57fa640 10 client.?.objecter ms_handle_connect 0x7f6cb80a9c00 2024-03-29T11:37:03.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.743+0000 7f6cb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:03.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.743+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13882409 0 0) 0x7f6cb80aeff0 con 0x7f6cb80a9c00 2024-03-29T11:37:03.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.743+0000 7f6cb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:03.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.743+0000 7f6cb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:03.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.743+0000 7f6cb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:03.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.743+0000 7f6cb57fa640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:37:03.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.743+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c9c004eb0 con 0x7f6cb80a9c00 2024-03-29T11:37:03.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 10 client.?.objecter ms_handle_connect 0x7f6cb80b2800 2024-03-29T11:37:03.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 278884633 0 0) 0x7f6c9c004eb0 con 0x7f6cb80a9c00 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb80aeff0 con 0x7f6cb80a9c00 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6cac001950 con 0x7f6cb80a9c00 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 10 monclient(hunting): dump: 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:03.744 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:03.745 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:03.745 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:03.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:03.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2128802781 0 0) 0x7f6cb80aeff0 con 0x7f6cb80a9c00 2024-03-29T11:37:03.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 >> v1:172.21.15.100:6790/0 conn(0x7f6cb80b38c0 legacy=0x7f6cb81d6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:03.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 >> v1:172.21.15.100:6789/0 conn(0x7f6cb80b2800 legacy=0x7f6cb8139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:03.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:03.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 1 monclient: found mon.b 2024-03-29T11:37:03.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:03.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:03.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb813a680 con 0x7f6cb80a9c00 2024-03-29T11:37:03.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.744+0000 7f6cb57fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:03.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.745+0000 7f6cb57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.745+0000 7f6cb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.745+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cac001c50 con 0x7f6cb80a9c00 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.745+0000 7f6cb57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6cac003a60 con 0x7f6cb80a9c00 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cb6ffd640 4 set_mon_vals no callback set 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cb57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cb57fa640 10 monclient: dump: 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:03.748 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cbdf80640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cbdf80640 10 monclient: _renew_subs 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cbdf80640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cbdf80640 1 -- 172.21.15.100:0/751330700 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb813a8d0 con 0x7f6cb80a9c00 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cbdf80640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cbdf80640 10 monclient: _renew_subs 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cbdf80640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cbdf80640 1 -- 172.21.15.100:0/751330700 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb813e800 con 0x7f6cb80a9c00 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.746+0000 7f6cbdf80640 1 librados: init done 2024-03-29T11:37:03.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.747+0000 7f6ca2ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:03.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.747+0000 7f6ca2ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:03.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.748+0000 7f6ca2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:03.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.748+0000 7f6ca2ffd640 1 -- 172.21.15.100:0/751330700 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c90005080 con 0x7f6cb80a9c00 2024-03-29T11:37:03.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.750+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f6cac003bc0 con 0x7f6cb80a9c00 2024-03-29T11:37:03.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.752+0000 7f6cb7fff640 10 monclient: get_auth_request con 0x7f6c9c06dcc0 auth_method 0 2024-03-29T11:37:03.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.752+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f6cac084b50 con 0x7f6cb80a9c00 2024-03-29T11:37:03.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.752+0000 7f6cb57fa640 10 client.4261.objecter ms_dispatch 0x7f6cb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:03.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.752+0000 7f6cb57fa640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:03.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.752+0000 7f6cb57fa640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:03.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.752+0000 7f6cb57fa640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:37:03.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.752+0000 7f6cb57fa640 10 client.4261.objecter ms_handle_connect 0x7f6c9c06dcc0 2024-03-29T11:37:03.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.757+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6cac050430 con 0x7f6cb80a9c00 2024-03-29T11:37:03.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.757+0000 7f6cb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:03.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:03.757+0000 7f6cb57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:04.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.083+0000 7f6ca2ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:04.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.083+0000 7f6ca2ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:04.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.083+0000 7f6ca2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:04.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.084+0000 7f6ca2ffd640 1 -- 172.21.15.100:0/751330700 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c90005330 con 0x7f6cb80a9c00 2024-03-29T11:37:04.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.087+0000 7f6cb57fa640 1 -- 172.21.15.100:0/751330700 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f6cac052360 con 0x7f6cb80a9c00 2024-03-29T11:37:04.086 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:37:04.087 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:04.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.087+0000 7f6cb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:04.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.087+0000 7f6cb57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:04.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.090+0000 7f6ca2ffd640 10 librados: watch_flush enter 2024-03-29T11:37:04.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.090+0000 7f6ca2ffd640 10 librados: watch_flush exit 2024-03-29T11:37:04.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.090+0000 7f6ca2ffd640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:37:04.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.090+0000 7f6ca2ffd640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:37:04.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.090+0000 7f6ca2ffd640 1 -- 172.21.15.100:0/751330700 >> v1:172.21.15.100:6816/3220289657 conn(0x7f6c9c06dcc0 legacy=0x7f6c9c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:04.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.090+0000 7f6ca2ffd640 10 monclient: shutdown 2024-03-29T11:37:04.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.090+0000 7f6ca2ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:04.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.090+0000 7f6ca2ffd640 1 -- 172.21.15.100:0/751330700 >> v1:172.21.15.191:6789/0 conn(0x7f6cb80a9c00 legacy=0x7f6cb80ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:04.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.094+0000 7f6ca2ffd640 1 -- 172.21.15.100:0/751330700 shutdown_connections 2024-03-29T11:37:04.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.094+0000 7f6ca2ffd640 1 -- 172.21.15.100:0/751330700 >> 172.21.15.100:0/751330700 conn(0x7f6cb8066a00 msgr2=0x7f6cb8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:04.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.094+0000 7f6ca2ffd640 1 -- 172.21.15.100:0/751330700 shutdown_connections 2024-03-29T11:37:04.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.094+0000 7f6ca2ffd640 1 -- 172.21.15.100:0/751330700 wait complete. 2024-03-29T11:37:04.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:04.094+0000 7f6ca2ffd640 1 librados: shutdown 2024-03-29T11:37:04.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:10.918 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:11.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.070+0000 7f21bbbf6640 10 monclient: get_monmap_and_config 2024-03-29T11:37:11.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.071+0000 7f21bbbf6640 10 monclient: build_initial_monmap 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.071+0000 7f21bbbf6640 10 monclient: monmap: 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:37:11.072043+0000 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:37:11.072043+0000 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:37:11.071 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:11.074 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.072+0000 7f21bbbf6640 1 Processor -- start 2024-03-29T11:37:11.074 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.072+0000 7f21bbbf6640 1 -- start start 2024-03-29T11:37:11.074 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.072+0000 7f21bbbf6640 10 monclient: init 2024-03-29T11:37:11.074 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.073+0000 7f21bbbf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:11.075 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.073+0000 7f21bbbf6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:11.075 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.073+0000 7f21bbbf6640 10 monclient(hunting): picked mon.noname-b con 0x7f21b40b1f60 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:11.075 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.073+0000 7f21bbbf6640 10 monclient(hunting): picked mon.noname-a con 0x7f21b4071eb0 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:11.075 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.073+0000 7f21bbbf6640 10 monclient(hunting): picked mon.noname-c con 0x7f21b4143d10 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:11.075 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.073+0000 7f21bbbf6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b406ef80 con 0x7f21b4071eb0 2024-03-29T11:37:11.075 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.073+0000 7f21bbbf6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b406f260 con 0x7f21b4143d10 2024-03-29T11:37:11.075 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.073+0000 7f21bbbf6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b40b40c0 con 0x7f21b40b1f60 2024-03-29T11:37:11.075 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.073+0000 7f21bbbf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:11.075 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.073+0000 7f21bbbf6640 10 monclient(hunting): authenticate will time out at 951.268416s 2024-03-29T11:37:11.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.076+0000 7f21b9992640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f21b40b1f60 0x7f21b4143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:46007/0 (socket says 172.21.15.100:46007) 2024-03-29T11:37:11.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.076+0000 7f21b9992640 1 -- 172.21.15.100:0/238077250 learned_addr learned my addr 172.21.15.100:0/238077250 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:11.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.079+0000 7f21b8990640 1 -- 172.21.15.100:0/238077250 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116412420 0 0) 0x7f21b40b40c0 con 0x7f21b40b1f60 2024-03-29T11:37:11.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.079+0000 7f21b8990640 10 monclient(hunting): _negotiate 2024-03-29T11:37:11.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.079+0000 7f21b8990640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:11.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.079+0000 7f21b8990640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:11.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.079+0000 7f21b8990640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:37:11.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.079+0000 7f21b8990640 1 -- 172.21.15.100:0/238077250 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21a0003520 con 0x7f21b40b1f60 2024-03-29T11:37:11.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 1 -- 172.21.15.100:0/238077250 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2540527222 0 0) 0x7f21a0003520 con 0x7f21b40b1f60 2024-03-29T11:37:11.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 1 -- 172.21.15.100:0/238077250 >> v1:172.21.15.191:6789/0 conn(0x7f21b4143d10 legacy=0x7f21b4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:11.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 1 -- 172.21.15.100:0/238077250 >> v1:172.21.15.100:6789/0 conn(0x7f21b4071eb0 legacy=0x7f21b4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:11.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 10 monclient: _finish_hunting 0 2024-03-29T11:37:11.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 1 monclient: found mon.noname-b 2024-03-29T11:37:11.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:11.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:37:11.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 1 -- 172.21.15.100:0/238077250 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b4155d70 con 0x7f21b40b1f60 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 10 monclient: _finish_auth 0 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 10 monclient: _check_auth_tickets 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 1 -- 172.21.15.100:0/238077250 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f21b0001a10 con 0x7f21b40b1f60 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 10 monclient: dump: 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:11.086 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21bbbf6640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21bbbf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 1 -- 172.21.15.100:0/238077250 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21b0001be0 con 0x7f21b40b1f60 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21bbbf6640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21bbbf6640 4 set_mon_vals no callback set 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.083+0000 7f21b8990640 1 -- 172.21.15.100:0/238077250 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f21b0002030 con 0x7f21b40b1f60 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.084+0000 7f21b8990640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.084+0000 7f21b8990640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.084+0000 7f21b8990640 10 monclient: dump: 2024-03-29T11:37:11.087 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.084+0000 7f21bbbf6640 10 monclient: shutdown 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.084+0000 7f21bbbf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.084+0000 7f21bbbf6640 1 -- 172.21.15.100:0/238077250 >> v1:172.21.15.100:6790/0 conn(0x7f21b40b1f60 legacy=0x7f21b4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:11.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.084+0000 7f21bbbf6640 1 -- 172.21.15.100:0/238077250 shutdown_connections 2024-03-29T11:37:11.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.084+0000 7f21bbbf6640 1 -- 172.21.15.100:0/238077250 >> 172.21.15.100:0/238077250 conn(0x7f21b4066a00 msgr2=0x7f21b4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:11.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.084+0000 7f21bbbf6640 1 -- 172.21.15.100:0/238077250 shutdown_connections 2024-03-29T11:37:11.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.084+0000 7f21bbbf6640 1 -- 172.21.15.100:0/238077250 wait complete. 2024-03-29T11:37:11.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.085+0000 7f21bbbf6640 10 monclient: build_initial_monmap 2024-03-29T11:37:11.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.085+0000 7f21bbbf6640 10 monclient: monmap: 2024-03-29T11:37:11.089 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:11.089 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:11.089 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.086+0000 7f21bbbf6640 1 librados: starting msgr at 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.086+0000 7f21bbbf6640 1 librados: starting objecter 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.086+0000 7f21bbbf6640 1 Processor -- start 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.086+0000 7f21bbbf6640 1 -- start start 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.086+0000 7f21bbbf6640 1 librados: setting wanted keys 2024-03-29T11:37:11.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.086+0000 7f21bbbf6640 1 librados: calling monclient init 2024-03-29T11:37:11.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.086+0000 7f21bbbf6640 10 monclient: init 2024-03-29T11:37:11.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.087+0000 7f21bbbf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:11.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.087+0000 7f21bbbf6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:11.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.087+0000 7f21bbbf6640 10 monclient(hunting): picked mon.noname-b con 0x7f21b40b1f60 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:11.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.087+0000 7f21bbbf6640 10 monclient(hunting): picked mon.noname-a con 0x7f21b4143d10 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:11.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.087+0000 7f21bbbf6640 10 monclient(hunting): picked mon.noname-c con 0x7f21b4136cd0 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:11.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.087+0000 7f21bbbf6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b4153e40 con 0x7f21b4143d10 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.087+0000 7f21bbbf6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b4154c70 con 0x7f21b40b1f60 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.087+0000 7f21bbbf6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b41427a0 con 0x7f21b4136cd0 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.087+0000 7f21bbbf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.087+0000 7f21bbbf6640 10 monclient(hunting): authenticate will time out at 951.282342s 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.093+0000 7f21b9191640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f21b4143d10 0x7f21b413f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:36095/0 (socket says 172.21.15.100:36095) 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.093+0000 7f21b9191640 1 -- 172.21.15.100:0/3564662970 learned_addr learned my addr 172.21.15.100:0/3564662970 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.093+0000 7f21aa7fc640 10 client.?.objecter ms_handle_connect 0x7f21b4143d10 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.093+0000 7f21aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 client.?.objecter ms_handle_connect 0x7f21b4136cd0 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 client.?.objecter ms_handle_connect 0x7f21b40b1f60 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166565736 0 0) 0x7f21b41427a0 con 0x7f21b4136cd0 2024-03-29T11:37:11.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2198004eb0 con 0x7f21b4136cd0 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692935901 0 0) 0x7f21b4154c70 con 0x7f21b40b1f60 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b41427a0 con 0x7f21b40b1f60 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 583688326 0 0) 0x7f2198004eb0 con 0x7f21b4136cd0 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21b4154c70 con 0x7f21b4136cd0 2024-03-29T11:37:11.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f21ac001750 con 0x7f21b4136cd0 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): dump: 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:11.104 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.094+0000 7f21aa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3982137549 0 0) 0x7f21b4154c70 con 0x7f21b4136cd0 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 >> v1:172.21.15.191:6789/0 conn(0x7f21b40b1f60 legacy=0x7f21b4153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 >> v1:172.21.15.100:6789/0 conn(0x7f21b4143d10 legacy=0x7f21b413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 1 monclient: found mon.c 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b4143030 con 0x7f21b4136cd0 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21bbbf6640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21bbbf6640 10 monclient: _renew_subs 2024-03-29T11:37:11.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21bbbf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21bbbf6640 1 -- 172.21.15.100:0/3564662970 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b41420a0 con 0x7f21b4136cd0 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21bbbf6640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21bbbf6640 10 monclient: _renew_subs 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21bbbf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21bbbf6640 1 -- 172.21.15.100:0/3564662970 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b41555c0 con 0x7f21b4136cd0 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.095+0000 7f21bbbf6640 1 librados: init done 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.096+0000 7f21bbbf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.096+0000 7f21bbbf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.096+0000 7f21bbbf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.096+0000 7f21bbbf6640 1 -- 172.21.15.100:0/3564662970 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b4137110 con 0x7f21b4136cd0 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.097+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21ac001c60 con 0x7f21b4136cd0 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.097+0000 7f21aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:11.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.097+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f21ac003790 con 0x7f21b4136cd0 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.097+0000 7f21aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.097+0000 7f21aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.097+0000 7f21aa7fc640 10 monclient: dump: 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:11.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.097+0000 7f21ab7fe640 4 set_mon_vals no callback set 2024-03-29T11:37:11.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.097+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f21ac0039b0 con 0x7f21b4136cd0 2024-03-29T11:37:11.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.098+0000 7f21b9992640 10 monclient: get_auth_request con 0x7f219806de00 auth_method 0 2024-03-29T11:37:11.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.098+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f21ac083740 con 0x7f21b4136cd0 2024-03-29T11:37:11.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.098+0000 7f21aa7fc640 10 client.4256.objecter ms_dispatch 0x7f21b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:11.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.098+0000 7f21aa7fc640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:11.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.098+0000 7f21aa7fc640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:11.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.098+0000 7f21aa7fc640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-29T11:37:11.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.098+0000 7f21aa7fc640 10 client.4256.objecter ms_handle_connect 0x7f219806de00 2024-03-29T11:37:11.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.109+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f21ac04f020 con 0x7f21b4136cd0 2024-03-29T11:37:11.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.109+0000 7f21aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:11.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.109+0000 7f21aa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:11.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.431+0000 7f21bbbf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:11.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.431+0000 7f21bbbf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:11.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.431+0000 7f21bbbf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:37:11.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.431+0000 7f21bbbf6640 1 -- 172.21.15.100:0/3564662970 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21b4075f10 con 0x7f21b4136cd0 2024-03-29T11:37:11.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.437+0000 7f21aa7fc640 1 -- 172.21.15.100:0/3564662970 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f21ac050f50 con 0x7f21b4136cd0 2024-03-29T11:37:11.440 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.437+0000 7f21aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:11.440 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.437+0000 7f21aa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:11.441 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:37:11.441 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:11.442 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.441+0000 7f21bbbf6640 10 librados: watch_flush enter 2024-03-29T11:37:11.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.442+0000 7f21bbbf6640 10 librados: watch_flush exit 2024-03-29T11:37:11.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.442+0000 7f21bbbf6640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-29T11:37:11.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.442+0000 7f21bbbf6640 10 client.4256.objecter successfully canceled tick 2024-03-29T11:37:11.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.443+0000 7f21bbbf6640 1 -- 172.21.15.100:0/3564662970 >> v1:172.21.15.100:6816/3220289657 conn(0x7f219806de00 legacy=0x7f21980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:11.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.443+0000 7f21bbbf6640 10 monclient: shutdown 2024-03-29T11:37:11.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.443+0000 7f21bbbf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:11.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.443+0000 7f21bbbf6640 1 -- 172.21.15.100:0/3564662970 >> v1:172.21.15.100:6790/0 conn(0x7f21b4136cd0 legacy=0x7f21b41e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:11.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.443+0000 7f21bbbf6640 1 -- 172.21.15.100:0/3564662970 shutdown_connections 2024-03-29T11:37:11.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.443+0000 7f21bbbf6640 1 -- 172.21.15.100:0/3564662970 >> 172.21.15.100:0/3564662970 conn(0x7f21b4066a00 msgr2=0x7f21b4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:11.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.443+0000 7f21bbbf6640 1 -- 172.21.15.100:0/3564662970 shutdown_connections 2024-03-29T11:37:11.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.443+0000 7f21bbbf6640 1 -- 172.21.15.100:0/3564662970 wait complete. 2024-03-29T11:37:11.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:11.443+0000 7f21bbbf6640 1 librados: shutdown 2024-03-29T11:37:11.461 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:18.268 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:18.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.420+0000 7fec3328d640 10 monclient: get_monmap_and_config 2024-03-29T11:37:18.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.420+0000 7fec3328d640 10 monclient: build_initial_monmap 2024-03-29T11:37:18.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.420+0000 7fec3328d640 10 monclient: monmap: 2024-03-29T11:37:18.421 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:18.421 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:18.421 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:37:18.421684+0000 2024-03-29T11:37:18.421 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:37:18.421684+0000 2024-03-29T11:37:18.422 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:18.422 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:18.422 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:18.422 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:37:18.422 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:37:18.422 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:18.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.422+0000 7fec3328d640 1 Processor -- start 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.422+0000 7fec3328d640 1 -- start start 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.422+0000 7fec3328d640 10 monclient: init 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.423+0000 7fec3328d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.423+0000 7fec3328d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.423+0000 7fec3328d640 10 monclient(hunting): picked mon.noname-a con 0x7fec2c0a9ac0 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.423+0000 7fec3328d640 10 monclient(hunting): picked mon.noname-c con 0x7fec2c13ee90 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.423+0000 7fec3328d640 10 monclient(hunting): picked mon.noname-b con 0x7fec2c0af900 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.423+0000 7fec3328d640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c0afd10 con 0x7fec2c0a9ac0 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.423+0000 7fec3328d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c0af6f0 con 0x7fec2c13ee90 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.423+0000 7fec3328d640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c0aac80 con 0x7fec2c0af900 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.423+0000 7fec3328d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:18.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.423+0000 7fec3328d640 10 monclient(hunting): authenticate will time out at 958.618294s 2024-03-29T11:37:18.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.427+0000 7fec3182a640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fec2c0af900 0x7fec2c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:51263/0 (socket says 172.21.15.100:51263) 2024-03-29T11:37:18.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.427+0000 7fec3182a640 1 -- 172.21.15.100:0/3573562122 learned_addr learned my addr 172.21.15.100:0/3573562122 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:18.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.427+0000 7fec2bfff640 1 -- 172.21.15.100:0/3573562122 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977560100 0 0) 0x7fec2c0aac80 con 0x7fec2c0af900 2024-03-29T11:37:18.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.427+0000 7fec2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:18.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.427+0000 7fec2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:18.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.427+0000 7fec2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:18.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.427+0000 7fec2bfff640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:37:18.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.427+0000 7fec2bfff640 1 -- 172.21.15.100:0/3573562122 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec14003520 con 0x7fec2c0af900 2024-03-29T11:37:18.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.428+0000 7fec2bfff640 1 -- 172.21.15.100:0/3573562122 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1518950472 0 0) 0x7fec14003520 con 0x7fec2c0af900 2024-03-29T11:37:18.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.428+0000 7fec2bfff640 1 -- 172.21.15.100:0/3573562122 >> v1:172.21.15.191:6789/0 conn(0x7fec2c13ee90 legacy=0x7fec2c13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:18.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.430+0000 7fec2bfff640 1 -- 172.21.15.100:0/3573562122 >> v1:172.21.15.100:6789/0 conn(0x7fec2c0a9ac0 legacy=0x7fec2c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:18.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.433+0000 7fec2bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:18.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.433+0000 7fec2bfff640 1 monclient: found mon.noname-b 2024-03-29T11:37:18.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.433+0000 7fec2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:18.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.433+0000 7fec2bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:37:18.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.433+0000 7fec2bfff640 1 -- 172.21.15.100:0/3573562122 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c0aaf60 con 0x7fec2c0af900 2024-03-29T11:37:18.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.433+0000 7fec2bfff640 10 monclient: _finish_auth 0 2024-03-29T11:37:18.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.433+0000 7fec2bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.433+0000 7fec2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec3328d640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec3328d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec2bfff640 1 -- 172.21.15.100:0/3573562122 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fec24001690 con 0x7fec2c0af900 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec2bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec2bfff640 10 monclient: dump: 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:18.435 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:18.436 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:18.436 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:18.436 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:18.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec3328d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:18.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec2bfff640 1 -- 172.21.15.100:0/3573562122 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec24001920 con 0x7fec2c0af900 2024-03-29T11:37:18.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:18.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec3328d640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:18.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.434+0000 7fec3328d640 4 set_mon_vals no callback set 2024-03-29T11:37:18.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.435+0000 7fec2bfff640 1 -- 172.21.15.100:0/3573562122 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fec24001cb0 con 0x7fec2c0af900 2024-03-29T11:37:18.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.435+0000 7fec2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:18.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.435+0000 7fec2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.435+0000 7fec2bfff640 10 monclient: dump: 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.435+0000 7fec3328d640 10 monclient: shutdown 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.435+0000 7fec3328d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:18.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.435+0000 7fec3328d640 1 -- 172.21.15.100:0/3573562122 >> v1:172.21.15.100:6790/0 conn(0x7fec2c0af900 legacy=0x7fec2c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:18.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.435+0000 7fec3328d640 1 -- 172.21.15.100:0/3573562122 shutdown_connections 2024-03-29T11:37:18.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.435+0000 7fec3328d640 1 -- 172.21.15.100:0/3573562122 >> 172.21.15.100:0/3573562122 conn(0x7fec2c0b35e0 msgr2=0x7fec2c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:18.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.439+0000 7fec3328d640 1 -- 172.21.15.100:0/3573562122 shutdown_connections 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.442+0000 7fec3328d640 1 -- 172.21.15.100:0/3573562122 wait complete. 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.443+0000 7fec3328d640 10 monclient: build_initial_monmap 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.443+0000 7fec3328d640 10 monclient: monmap: 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:37:18.448 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.443+0000 7fec3328d640 1 librados: starting msgr at 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.443+0000 7fec3328d640 1 librados: starting objecter 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.443+0000 7fec3328d640 1 Processor -- start 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.444+0000 7fec3328d640 1 -- start start 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.444+0000 7fec3328d640 1 librados: setting wanted keys 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.444+0000 7fec3328d640 1 librados: calling monclient init 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.444+0000 7fec3328d640 10 monclient: init 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.445+0000 7fec3328d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.445+0000 7fec3328d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.445+0000 7fec3328d640 10 monclient(hunting): picked mon.noname-c con 0x7fec2c0a9ac0 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.445+0000 7fec3328d640 10 monclient(hunting): picked mon.noname-b con 0x7fec2c0af900 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.445+0000 7fec3328d640 10 monclient(hunting): picked mon.noname-a con 0x7fec2c13ee90 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:18.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.445+0000 7fec3328d640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c142320 con 0x7fec2c13ee90 2024-03-29T11:37:18.450 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.445+0000 7fec3328d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c142590 con 0x7fec2c0af900 2024-03-29T11:37:18.450 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.445+0000 7fec3328d640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c1403e0 con 0x7fec2c0a9ac0 2024-03-29T11:37:18.450 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.445+0000 7fec3328d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:18.450 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.445+0000 7fec3328d640 10 monclient(hunting): authenticate will time out at 958.640579s 2024-03-29T11:37:18.451 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.451+0000 7fec30828640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fec2c0af900 0x7fec2c140a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:48767/0 (socket says 172.21.15.100:48767) 2024-03-29T11:37:18.451 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.451+0000 7fec30828640 1 -- 172.21.15.100:0/523334911 learned_addr learned my addr 172.21.15.100:0/523334911 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:18.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.453+0000 7fec29ffb640 10 client.?.objecter ms_handle_connect 0x7fec2c0af900 2024-03-29T11:37:18.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.453+0000 7fec29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:18.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.453+0000 7fec29ffb640 10 client.?.objecter ms_handle_connect 0x7fec2c13ee90 2024-03-29T11:37:18.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.453+0000 7fec29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:18.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.453+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001938266 0 0) 0x7fec2c142320 con 0x7fec2c13ee90 2024-03-29T11:37:18.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.453+0000 7fec29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:18.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.453+0000 7fec29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:18.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.453+0000 7fec29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:18.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.453+0000 7fec29ffb640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:37:18.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.453+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec18004eb0 con 0x7fec2c13ee90 2024-03-29T11:37:18.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.455+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608483110 0 0) 0x7fec2c142590 con 0x7fec2c0af900 2024-03-29T11:37:18.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.455+0000 7fec29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:18.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.455+0000 7fec29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:18.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.455+0000 7fec29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:18.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.455+0000 7fec29ffb640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:37:18.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.455+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c142320 con 0x7fec2c0af900 2024-03-29T11:37:18.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.455+0000 7fec29ffb640 10 client.?.objecter ms_handle_connect 0x7fec2c0a9ac0 2024-03-29T11:37:18.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.455+0000 7fec29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:18.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060797292 0 0) 0x7fec2c1403e0 con 0x7fec2c0a9ac0 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c142590 con 0x7fec2c0a9ac0 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3489309599 0 0) 0x7fec18004eb0 con 0x7fec2c13ee90 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec2c1403e0 con 0x7fec2c13ee90 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fec24001550 con 0x7fec2c13ee90 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 10 monclient(hunting): dump: 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:18.456 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.457 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.457 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:18.457 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:18.457 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:18.457 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:18.457 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:18.457 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:18.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.456+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1983539208 0 0) 0x7fec2c142590 con 0x7fec2c0a9ac0 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec18004eb0 con 0x7fec2c0a9ac0 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fec200017e0 con 0x7fec2c0a9ac0 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 10 monclient(hunting): dump: 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.458 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3258826866 0 0) 0x7fec2c142320 con 0x7fec2c0af900 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec2c142590 con 0x7fec2c0af900 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fec1c001b10 con 0x7fec2c0af900 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 10 monclient(hunting): dump: 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:18.459 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3653530626 0 0) 0x7fec2c1403e0 con 0x7fec2c13ee90 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 >> v1:172.21.15.100:6790/0 conn(0x7fec2c0a9ac0 legacy=0x7fec2c14b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 >> v1:172.21.15.191:6789/0 conn(0x7fec2c0af900 legacy=0x7fec2c140a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 1 monclient: found mon.a 2024-03-29T11:37:18.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c06e910 con 0x7fec2c13ee90 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec3328d640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec3328d640 10 monclient: _renew_subs 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec3328d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec3328d640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c130190 con 0x7fec2c13ee90 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec3328d640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec3328d640 10 monclient: _renew_subs 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec3328d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec3328d640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c1400a0 con 0x7fec2c13ee90 2024-03-29T11:37:18.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.457+0000 7fec3328d640 1 librados: init done 2024-03-29T11:37:18.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.459+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec24001730 con 0x7fec2c13ee90 2024-03-29T11:37:18.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.459+0000 7fec29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:18.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.459+0000 7fec2b7fe640 4 set_mon_vals no callback set 2024-03-29T11:37:18.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.460+0000 7fec3328d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:18.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.460+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fec24001250 con 0x7fec2c13ee90 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.460+0000 7fec3328d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.461+0000 7fec3328d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.461+0000 7fec3328d640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0c005080 con 0x7fec2c13ee90 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.461+0000 7fec29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.461+0000 7fec29ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.461+0000 7fec29ffb640 10 monclient: dump: 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:18.464 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:18.465 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:18.465 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:18.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.461+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fec2401ac00 con 0x7fec2c13ee90 2024-03-29T11:37:18.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.462+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fec24085b90 con 0x7fec2c13ee90 2024-03-29T11:37:18.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.462+0000 7fec29ffb640 10 client.4269.objecter ms_dispatch 0x7fec2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:18.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.462+0000 7fec29ffb640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:18.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.462+0000 7fec29ffb640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:18.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.462+0000 7fec29ffb640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T11:37:18.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.462+0000 7fec31029640 10 monclient: get_auth_request con 0x7fec1806dfe0 auth_method 0 2024-03-29T11:37:18.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.463+0000 7fec29ffb640 10 client.4269.objecter ms_handle_connect 0x7fec1806dfe0 2024-03-29T11:37:18.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.470+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec240502f0 con 0x7fec2c13ee90 2024-03-29T11:37:18.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.470+0000 7fec29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:18.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.470+0000 7fec29ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:18.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.793+0000 7fec3328d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:18.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.793+0000 7fec3328d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:18.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.793+0000 7fec3328d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:18.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.793+0000 7fec3328d640 1 -- 172.21.15.100:0/523334911 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec0c005330 con 0x7fec2c13ee90 2024-03-29T11:37:18.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.799+0000 7fec29ffb640 1 -- 172.21.15.100:0/523334911 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fec240523f0 con 0x7fec2c13ee90 2024-03-29T11:37:18.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.799+0000 7fec29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:18.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.799+0000 7fec29ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:18.802 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:37:18.802 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:18.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.805+0000 7fec3328d640 10 librados: watch_flush enter 2024-03-29T11:37:18.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.805+0000 7fec3328d640 10 librados: watch_flush exit 2024-03-29T11:37:18.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.805+0000 7fec3328d640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T11:37:18.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.805+0000 7fec3328d640 10 client.4269.objecter successfully canceled tick 2024-03-29T11:37:18.808 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.808+0000 7fec3328d640 1 -- 172.21.15.100:0/523334911 >> v1:172.21.15.100:6816/3220289657 conn(0x7fec1806dfe0 legacy=0x7fec180704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:18.808 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.808+0000 7fec3328d640 10 monclient: shutdown 2024-03-29T11:37:18.808 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.808+0000 7fec3328d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:18.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.808+0000 7fec3328d640 1 -- 172.21.15.100:0/523334911 >> v1:172.21.15.100:6789/0 conn(0x7fec2c13ee90 legacy=0x7fec2c1e69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:18.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.808+0000 7fec3328d640 1 -- 172.21.15.100:0/523334911 shutdown_connections 2024-03-29T11:37:18.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.808+0000 7fec3328d640 1 -- 172.21.15.100:0/523334911 >> 172.21.15.100:0/523334911 conn(0x7fec2c0b35e0 msgr2=0x7fec2c0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:18.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.809+0000 7fec3328d640 1 -- 172.21.15.100:0/523334911 shutdown_connections 2024-03-29T11:37:18.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.809+0000 7fec3328d640 1 -- 172.21.15.100:0/523334911 wait complete. 2024-03-29T11:37:18.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:18.809+0000 7fec3328d640 1 librados: shutdown 2024-03-29T11:37:18.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:25.632 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:25.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.782+0000 7fb90e57c640 10 monclient: get_monmap_and_config 2024-03-29T11:37:25.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.782+0000 7fb90e57c640 10 monclient: build_initial_monmap 2024-03-29T11:37:25.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.782+0000 7fb90e57c640 10 monclient: monmap: 2024-03-29T11:37:25.782 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:25.782 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:25.782 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:37:25.783683+0000 2024-03-29T11:37:25.782 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:37:25.783683+0000 2024-03-29T11:37:25.783 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:25.783 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:25.783 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:25.783 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:37:25.783 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:37:25.783 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:25.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.786+0000 7fb90e57c640 1 Processor -- start 2024-03-29T11:37:25.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.786+0000 7fb90e57c640 1 -- start start 2024-03-29T11:37:25.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.786+0000 7fb90e57c640 10 monclient: init 2024-03-29T11:37:25.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.787+0000 7fb90e57c640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:25.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.787+0000 7fb90e57c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:25.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.787+0000 7fb90e57c640 10 monclient(hunting): picked mon.noname-b con 0x7fb9081422b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:25.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.787+0000 7fb90e57c640 10 monclient(hunting): picked mon.noname-c con 0x7fb908142690 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:25.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.787+0000 7fb90e57c640 10 monclient(hunting): picked mon.noname-a con 0x7fb9080b17d0 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:25.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.787+0000 7fb90e57c640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb908142c80 con 0x7fb9080b17d0 2024-03-29T11:37:25.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.787+0000 7fb90e57c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9080aae40 con 0x7fb908142690 2024-03-29T11:37:25.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.787+0000 7fb90e57c640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90813fbd0 con 0x7fb9081422b0 2024-03-29T11:37:25.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.787+0000 7fb90e57c640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:25.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.788+0000 7fb90e57c640 10 monclient(hunting): authenticate will time out at 965.983099s 2024-03-29T11:37:25.790 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.791+0000 7fb90cb19640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fb9080b17d0 0x7fb9080b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:60199/0 (socket says 172.21.15.100:60199) 2024-03-29T11:37:25.790 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.791+0000 7fb90cb19640 1 -- 172.21.15.100:0/2919213160 learned_addr learned my addr 172.21.15.100:0/2919213160 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:25.792 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.792+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765040928 0 0) 0x7fb908142c80 con 0x7fb9080b17d0 2024-03-29T11:37:25.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:25.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:25.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:25.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:37:25.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f0003520 con 0x7fb9080b17d0 2024-03-29T11:37:25.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742537347 0 0) 0x7fb9080aae40 con 0x7fb908142690 2024-03-29T11:37:25.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:25.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:25.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.793+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb908142c80 con 0x7fb908142690 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1553754912 0 0) 0x7fb908142c80 con 0x7fb908142690 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 >> v1:172.21.15.100:6790/0 conn(0x7fb9081422b0 legacy=0x7fb90813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 >> v1:172.21.15.100:6789/0 conn(0x7fb9080b17d0 legacy=0x7fb9080b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 1 monclient: found mon.noname-c 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb908151940 con 0x7fb908142690 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.794+0000 7fb906ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:25.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb9000019f0 con 0x7fb908142690 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb90e57c640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb90e57c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb906ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb906ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb906ffd640 10 monclient: dump: 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:25.798 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb90e57c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb900001c10 con 0x7fb908142690 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb906ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb90e57c640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb90e57c640 4 set_mon_vals no callback set 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb906ffd640 1 -- 172.21.15.100:0/2919213160 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb900001fe0 con 0x7fb908142690 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb906ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.795+0000 7fb906ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.796+0000 7fb90e57c640 10 monclient: shutdown 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.796+0000 7fb906ffd640 10 monclient: dump: 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:25.799 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.796+0000 7fb90e57c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.796+0000 7fb90e57c640 1 -- 172.21.15.100:0/2919213160 >> v1:172.21.15.191:6789/0 conn(0x7fb908142690 legacy=0x7fb9080b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.796+0000 7fb90e57c640 1 -- 172.21.15.100:0/2919213160 shutdown_connections 2024-03-29T11:37:25.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.796+0000 7fb90e57c640 1 -- 172.21.15.100:0/2919213160 >> 172.21.15.100:0/2919213160 conn(0x7fb908066a00 msgr2=0x7fb908066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:25.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.797+0000 7fb90e57c640 1 -- 172.21.15.100:0/2919213160 shutdown_connections 2024-03-29T11:37:25.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.797+0000 7fb90e57c640 1 -- 172.21.15.100:0/2919213160 wait complete. 2024-03-29T11:37:25.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.798+0000 7fb90e57c640 10 monclient: build_initial_monmap 2024-03-29T11:37:25.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.798+0000 7fb90e57c640 10 monclient: monmap: 2024-03-29T11:37:25.802 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:25.802 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:25.802 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:37:25.802 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:37:25.802 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.799+0000 7fb90e57c640 1 librados: starting msgr at 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.799+0000 7fb90e57c640 1 librados: starting objecter 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.800+0000 7fb90e57c640 1 Processor -- start 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.800+0000 7fb90e57c640 1 -- start start 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.800+0000 7fb90e57c640 1 librados: setting wanted keys 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.800+0000 7fb90e57c640 1 librados: calling monclient init 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.800+0000 7fb90e57c640 10 monclient: init 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.801+0000 7fb90e57c640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:25.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.801+0000 7fb90e57c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:25.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.805+0000 7fb90e57c640 10 monclient(hunting): picked mon.noname-b con 0x7fb9080b17d0 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:25.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.805+0000 7fb907fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fb9080b17d0 0x7fb9081daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:34275/0 (socket says 172.21.15.100:34275) 2024-03-29T11:37:25.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.805+0000 7fb907fff640 1 -- 172.21.15.100:0/2160631706 learned_addr learned my addr 172.21.15.100:0/2160631706 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:25.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.807+0000 7fb90e57c640 10 monclient(hunting): picked mon.noname-a con 0x7fb9081422b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:25.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.809+0000 7fb904ff9640 10 client.?.objecter ms_handle_connect 0x7fb9080b17d0 2024-03-29T11:37:25.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.809+0000 7fb904ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:25.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.809+0000 7fb90e57c640 10 monclient(hunting): picked mon.noname-c con 0x7fb908142690 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:25.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.809+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9080abd80 con 0x7fb9081422b0 2024-03-29T11:37:25.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.809+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9080aba20 con 0x7fb9080b17d0 2024-03-29T11:37:25.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.809+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9080ac7a0 con 0x7fb908142690 2024-03-29T11:37:25.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.809+0000 7fb90e57c640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:25.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.809+0000 7fb90e57c640 10 monclient(hunting): authenticate will time out at 966.005036s 2024-03-29T11:37:25.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.813+0000 7fb904ff9640 10 client.?.objecter ms_handle_connect 0x7fb9081422b0 2024-03-29T11:37:25.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.813+0000 7fb904ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:25.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.813+0000 7fb904ff9640 10 client.?.objecter ms_handle_connect 0x7fb908142690 2024-03-29T11:37:25.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.813+0000 7fb904ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:25.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.813+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948905203 0 0) 0x7fb9080aba20 con 0x7fb9080b17d0 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.813+0000 7fb904ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.813+0000 7fb904ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.813+0000 7fb904ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.813+0000 7fb904ff9640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.813+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f4004eb0 con 0x7fb9080b17d0 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843148870 0 0) 0x7fb9080abd80 con 0x7fb9081422b0 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9080aba20 con 0x7fb9081422b0 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528755391 0 0) 0x7fb9080ac7a0 con 0x7fb908142690 2024-03-29T11:37:25.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9080abd80 con 0x7fb908142690 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1740379296 0 0) 0x7fb9080aba20 con 0x7fb9081422b0 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9080ac7a0 con 0x7fb9081422b0 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb9000018b0 con 0x7fb9081422b0 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.814+0000 7fb904ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:25.816 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:25.817 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:25.817 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:25.817 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:25.817 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:25.817 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:25.817 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:25.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:25.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 530243431 0 0) 0x7fb9080ac7a0 con 0x7fb9081422b0 2024-03-29T11:37:25.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 >> v1:172.21.15.100:6790/0 conn(0x7fb908142690 legacy=0x7fb9081e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:25.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 >> v1:172.21.15.191:6789/0 conn(0x7fb9080b17d0 legacy=0x7fb9081daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:25.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:25.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 1 monclient: found mon.a 2024-03-29T11:37:25.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:25.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:25.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9080aca80 con 0x7fb9081422b0 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb900003c20 con 0x7fb9081422b0 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb900003f30 con 0x7fb9081422b0 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.815+0000 7fb904ff9640 10 monclient: dump: 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:25.819 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.816+0000 7fb9067fc640 4 set_mon_vals no callback set 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.817+0000 7fb90e57c640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.817+0000 7fb90e57c640 10 monclient: _renew_subs 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.817+0000 7fb90e57c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.817+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9080acd60 con 0x7fb9081422b0 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.817+0000 7fb90e57c640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.817+0000 7fb90e57c640 10 monclient: _renew_subs 2024-03-29T11:37:25.820 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.817+0000 7fb90e57c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:25.821 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.817+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb908132380 con 0x7fb9081422b0 2024-03-29T11:37:25.821 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.817+0000 7fb90e57c640 1 librados: init done 2024-03-29T11:37:25.821 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.819+0000 7fb90e57c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:25.821 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.819+0000 7fb90e57c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:25.821 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.819+0000 7fb90e57c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:25.821 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.819+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb908075f10 con 0x7fb9081422b0 2024-03-29T11:37:25.822 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.822+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fb900019f00 con 0x7fb9081422b0 2024-03-29T11:37:25.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.823+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fb900083ba0 con 0x7fb9081422b0 2024-03-29T11:37:25.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.823+0000 7fb904ff9640 10 client.4275.objecter ms_dispatch 0x7fb908000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:25.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.823+0000 7fb904ff9640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:25.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.823+0000 7fb904ff9640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:25.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.823+0000 7fb904ff9640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-29T11:37:25.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.824+0000 7fb907fff640 10 monclient: get_auth_request con 0x7fb8f406de00 auth_method 0 2024-03-29T11:37:25.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.824+0000 7fb904ff9640 10 client.4275.objecter ms_handle_connect 0x7fb8f406de00 2024-03-29T11:37:25.831 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.830+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb90004f480 con 0x7fb9081422b0 2024-03-29T11:37:25.831 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.830+0000 7fb904ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:25.831 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:25.830+0000 7fb904ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:26.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.157+0000 7fb90e57c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:26.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.157+0000 7fb90e57c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:26.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.157+0000 7fb90e57c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:26.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.157+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9080b2270 con 0x7fb9081422b0 2024-03-29T11:37:26.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.158+0000 7fb904ff9640 1 -- 172.21.15.100:0/2160631706 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fb9000513b0 con 0x7fb9081422b0 2024-03-29T11:37:26.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.158+0000 7fb904ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:26.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.158+0000 7fb904ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:26.160 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:37:26.160 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:26.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.164+0000 7fb90e57c640 10 librados: watch_flush enter 2024-03-29T11:37:26.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.164+0000 7fb90e57c640 10 librados: watch_flush exit 2024-03-29T11:37:26.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.164+0000 7fb90e57c640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-29T11:37:26.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.164+0000 7fb90e57c640 10 client.4275.objecter successfully canceled tick 2024-03-29T11:37:26.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.165+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 >> v1:172.21.15.100:6816/3220289657 conn(0x7fb8f406de00 legacy=0x7fb8f40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:26.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.165+0000 7fb90e57c640 10 monclient: shutdown 2024-03-29T11:37:26.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.165+0000 7fb90e57c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:26.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.165+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 >> v1:172.21.15.100:6789/0 conn(0x7fb9081422b0 legacy=0x7fb9081db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:26.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.165+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 shutdown_connections 2024-03-29T11:37:26.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.165+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 >> 172.21.15.100:0/2160631706 conn(0x7fb908066a00 msgr2=0x7fb9080b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:26.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.168+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 shutdown_connections 2024-03-29T11:37:26.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.168+0000 7fb90e57c640 1 -- 172.21.15.100:0/2160631706 wait complete. 2024-03-29T11:37:26.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:26.168+0000 7fb90e57c640 1 librados: shutdown 2024-03-29T11:37:26.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:32.995 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:33.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.145+0000 7f11dad7a640 10 monclient: get_monmap_and_config 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.145+0000 7f11dad7a640 10 monclient: build_initial_monmap 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.146+0000 7f11dad7a640 10 monclient: monmap: 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:37:33.147007+0000 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:37:33.147007+0000 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:37:33.146 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:33.147 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.147+0000 7f11dad7a640 1 Processor -- start 2024-03-29T11:37:33.147 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.147+0000 7f11dad7a640 1 -- start start 2024-03-29T11:37:33.147 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.147+0000 7f11dad7a640 10 monclient: init 2024-03-29T11:37:33.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11dad7a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:33.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11dad7a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:33.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11dad7a640 10 monclient(hunting): picked mon.noname-c con 0x7f11d40b4710 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:33.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11dad7a640 10 monclient(hunting): picked mon.noname-b con 0x7f11d40b3970 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:33.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11dad7a640 10 monclient(hunting): picked mon.noname-a con 0x7f11d413f390 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:33.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11dad7a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d40af020 con 0x7f11d413f390 2024-03-29T11:37:33.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11dad7a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d40b0910 con 0x7f11d40b4710 2024-03-29T11:37:33.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11dad7a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d40b0bf0 con 0x7f11d40b3970 2024-03-29T11:37:33.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11dad7a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:33.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11dad7a640 10 monclient(hunting): authenticate will time out at 973.343557s 2024-03-29T11:37:33.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11d9317640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f11d413f390 0x7f11d413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:39892/0 (socket says 172.21.15.100:39892) 2024-03-29T11:37:33.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.148+0000 7f11d9317640 1 -- 172.21.15.100:0/3463904860 learned_addr learned my addr 172.21.15.100:0/3463904860 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427785875 0 0) 0x7f11d40b0910 con 0x7f11d40b4710 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c4003520 con 0x7f11d40b4710 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570718501 0 0) 0x7f11d40b0bf0 con 0x7f11d40b3970 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:33.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d40b0910 con 0x7f11d40b3970 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1741339087 0 0) 0x7f11c4003520 con 0x7f11d40b4710 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 >> v1:172.21.15.100:6790/0 conn(0x7f11d40b3970 legacy=0x7f11d40af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 >> v1:172.21.15.100:6789/0 conn(0x7f11d413f390 legacy=0x7f11d413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 1 monclient: found mon.noname-c 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40aeb00 con 0x7f11d40b4710 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f11c80015c0 con 0x7f11d40b4710 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:33.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11d37fe640 10 monclient: dump: 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11dad7a640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.149+0000 7f11dad7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:33.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.150+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11c80017e0 con 0x7f11d40b4710 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.150+0000 7f11d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.150+0000 7f11d37fe640 1 -- 172.21.15.100:0/3463904860 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f11c8001b70 con 0x7f11d40b4710 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.150+0000 7f11dad7a640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.150+0000 7f11dad7a640 4 set_mon_vals no callback set 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.150+0000 7f11d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.150+0000 7f11d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.150+0000 7f11d37fe640 10 monclient: dump: 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:33.154 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:33.155 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:33.155 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:33.155 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:33.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.151+0000 7f11dad7a640 10 monclient: shutdown 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.151+0000 7f11dad7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.151+0000 7f11dad7a640 1 -- 172.21.15.100:0/3463904860 >> v1:172.21.15.191:6789/0 conn(0x7f11d40b4710 legacy=0x7f11d40b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.151+0000 7f11dad7a640 1 -- 172.21.15.100:0/3463904860 shutdown_connections 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.151+0000 7f11dad7a640 1 -- 172.21.15.100:0/3463904860 >> 172.21.15.100:0/3463904860 conn(0x7f11d4066a00 msgr2=0x7f11d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.152+0000 7f11dad7a640 1 -- 172.21.15.100:0/3463904860 shutdown_connections 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.152+0000 7f11dad7a640 1 -- 172.21.15.100:0/3463904860 wait complete. 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.153+0000 7f11dad7a640 10 monclient: build_initial_monmap 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.153+0000 7f11dad7a640 10 monclient: monmap: 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:33.159 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.154+0000 7f11dad7a640 1 librados: starting msgr at 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.154+0000 7f11dad7a640 1 librados: starting objecter 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.154+0000 7f11dad7a640 1 Processor -- start 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.154+0000 7f11dad7a640 1 -- start start 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.154+0000 7f11dad7a640 1 librados: setting wanted keys 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.154+0000 7f11dad7a640 1 librados: calling monclient init 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.154+0000 7f11dad7a640 10 monclient: init 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.155+0000 7f11dad7a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.155+0000 7f11dad7a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.155+0000 7f11dad7a640 10 monclient(hunting): picked mon.noname-a con 0x7f11d40b3970 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:33.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.155+0000 7f11d8b16640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f11d40b3970 0x7f11d4142510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:40172/0 (socket says 172.21.15.100:40172) 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.155+0000 7f11dad7a640 10 monclient(hunting): picked mon.noname-c con 0x7f11d40b4710 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.155+0000 7f11d8b16640 1 -- 172.21.15.100:0/3133525751 learned_addr learned my addr 172.21.15.100:0/3133525751 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.155+0000 7f11dad7a640 10 monclient(hunting): picked mon.noname-b con 0x7f11d413f390 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.155+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d4138f00 con 0x7f11d40b3970 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.155+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d4136510 con 0x7f11d413f390 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d4143190 con 0x7f11d40b4710 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11dad7a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 10 client.?.objecter ms_handle_connect 0x7f11d40b3970 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11dad7a640 10 monclient(hunting): authenticate will time out at 973.351206s 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 10 client.?.objecter ms_handle_connect 0x7f11d413f390 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:33.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 10 client.?.objecter ms_handle_connect 0x7f11d40b4710 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899865564 0 0) 0x7f11d4138f00 con 0x7f11d40b3970 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.156+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b8004eb0 con 0x7f11d40b3970 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807163101 0 0) 0x7f11d4136510 con 0x7f11d413f390 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:37:33.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d4138f00 con 0x7f11d413f390 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162629192 0 0) 0x7f11d4143190 con 0x7f11d40b4710 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b8005360 con 0x7f11d40b4710 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3981317818 0 0) 0x7f11b8004eb0 con 0x7f11d40b3970 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d4143190 con 0x7f11d40b3970 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f11c8001bb0 con 0x7f11d40b3970 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): dump: 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:33.163 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 187894662 0 0) 0x7f11d4143190 con 0x7f11d40b3970 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 >> v1:172.21.15.100:6790/0 conn(0x7f11d40b4710 legacy=0x7f11d41400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 >> v1:172.21.15.191:6789/0 conn(0x7f11d413f390 legacy=0x7f11d41407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:33.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 monclient: found mon.a 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d4135040 con 0x7f11d40b3970 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11dad7a640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11dad7a640 10 monclient: _renew_subs 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11dad7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.157+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d4135290 con 0x7f11d40b3970 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11dad7a640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11dad7a640 10 monclient: _renew_subs 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11dad7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d41e37f0 con 0x7f11d40b3970 2024-03-29T11:37:33.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11dad7a640 1 librados: init done 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11c80038a0 con 0x7f11d40b3970 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f11c8003c30 con 0x7f11d40b3970 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11d2ffd640 4 set_mon_vals no callback set 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.158+0000 7f11d17fa640 10 monclient: dump: 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:33.166 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:33.167 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:33.167 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.159+0000 7f11dad7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.159+0000 7f11dad7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.160+0000 7f11dad7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.160+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b4005080 con 0x7f11d40b3970 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.161+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f11c8003d90 con 0x7f11d40b3970 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.162+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f11c8084b90 con 0x7f11d40b3970 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.162+0000 7f11d17fa640 10 client.4281.objecter ms_dispatch 0x7f11d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.162+0000 7f11d17fa640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.162+0000 7f11d17fa640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.162+0000 7f11d17fa640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.162+0000 7f11d3fff640 10 monclient: get_auth_request con 0x7f11b806dfd0 auth_method 0 2024-03-29T11:37:33.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.162+0000 7f11d17fa640 10 client.4281.objecter ms_handle_connect 0x7f11b806dfd0 2024-03-29T11:37:33.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.169+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11c8050470 con 0x7f11d40b3970 2024-03-29T11:37:33.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.169+0000 7f11d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:33.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.169+0000 7f11d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:33.539 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.539+0000 7f11dad7a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:33.539 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.539+0000 7f11dad7a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:33.539 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.539+0000 7f11dad7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:37:33.539 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.539+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11b4005330 con 0x7f11d40b3970 2024-03-29T11:37:33.540 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.540+0000 7f11d17fa640 1 -- 172.21.15.100:0/3133525751 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f11c80523a0 con 0x7f11d40b3970 2024-03-29T11:37:33.540 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.540+0000 7f11d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:33.540 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.540+0000 7f11d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:33.540 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:37:33.540 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:33.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 10 librados: watch_flush enter 2024-03-29T11:37:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 10 librados: watch_flush exit 2024-03-29T11:37:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-29T11:37:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 10 client.4281.objecter successfully canceled tick 2024-03-29T11:37:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 >> v1:172.21.15.100:6816/3220289657 conn(0x7f11b806dfd0 legacy=0x7f11b8070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 10 monclient: shutdown 2024-03-29T11:37:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 >> v1:172.21.15.100:6789/0 conn(0x7f11d40b3970 legacy=0x7f11d4142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 shutdown_connections 2024-03-29T11:37:33.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 >> 172.21.15.100:0/3133525751 conn(0x7f11d4066a00 msgr2=0x7f11d413f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:33.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 shutdown_connections 2024-03-29T11:37:33.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 1 -- 172.21.15.100:0/3133525751 wait complete. 2024-03-29T11:37:33.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:33.544+0000 7f11dad7a640 1 librados: shutdown 2024-03-29T11:37:33.559 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:40.365 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:40.524 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.524+0000 7f3452ac1640 10 monclient: get_monmap_and_config 2024-03-29T11:37:40.524 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.524+0000 7f3452ac1640 10 monclient: build_initial_monmap 2024-03-29T11:37:40.524 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.524+0000 7f3452ac1640 10 monclient: monmap: 2024-03-29T11:37:40.524 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:40.524 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:40.524 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:37:40.525404+0000 2024-03-29T11:37:40.524 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:37:40.525404+0000 2024-03-29T11:37:40.524 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:40.524 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:40.525 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:40.525 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:37:40.525 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:37:40.525 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.525+0000 7f3452ac1640 1 Processor -- start 2024-03-29T11:37:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.525+0000 7f3452ac1640 1 -- start start 2024-03-29T11:37:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.525+0000 7f3452ac1640 10 monclient: init 2024-03-29T11:37:40.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.526+0000 7f3452ac1640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.526+0000 7f3452ac1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.526+0000 7f3452ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f3444006e30 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.526+0000 7f3452ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f3444035e00 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.526+0000 7f3452ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f34440c9670 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.526+0000 7f3452ac1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3444035970 con 0x7f3444006e30 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.526+0000 7f3452ac1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3444036280 con 0x7f34440c9670 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.526+0000 7f3452ac1640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3444035640 con 0x7f3444035e00 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.526+0000 7f3452ac1640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.526+0000 7f3452ac1640 10 monclient(hunting): authenticate will time out at 980.721902s 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3451abf640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f3444006e30 0x7f34440cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:36102/0 (socket says 172.21.15.100:36102) 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3451abf640 1 -- 172.21.15.100:0/3569316384 learned_addr learned my addr 172.21.15.100:0/3569316384 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3450abd640 1 -- 172.21.15.100:0/3569316384 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015404238 0 0) 0x7f3444035970 con 0x7f3444006e30 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3450abd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:40.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3450abd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3450abd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3450abd640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:37:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3450abd640 1 -- 172.21.15.100:0/3569316384 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3438003520 con 0x7f3444006e30 2024-03-29T11:37:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3450abd640 1 -- 172.21.15.100:0/3569316384 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 234559078 0 0) 0x7f3438003520 con 0x7f3444006e30 2024-03-29T11:37:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3450abd640 1 -- 172.21.15.100:0/3569316384 >> v1:172.21.15.100:6790/0 conn(0x7f3444035e00 legacy=0x7f34440c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.527+0000 7f3450abd640 1 -- 172.21.15.100:0/3569316384 >> v1:172.21.15.191:6789/0 conn(0x7f34440c9670 legacy=0x7f34440d7560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:40.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:40.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 1 monclient: found mon.noname-a 2024-03-29T11:37:40.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:40.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:37:40.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 1 -- 172.21.15.100:0/3569316384 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444034e20 con 0x7f3444006e30 2024-03-29T11:37:40.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 10 monclient: _finish_auth 0 2024-03-29T11:37:40.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:40.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 1 -- 172.21.15.100:0/3569316384 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f344c04f730 con 0x7f3444006e30 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3452ac1640 5 monclient: authenticate success, global_id 4284 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3450abd640 10 monclient: dump: 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:40.532 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:40.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.530+0000 7f3452ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:40.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.531+0000 7f3450abd640 1 -- 172.21.15.100:0/3569316384 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f344c075f10 con 0x7f3444006e30 2024-03-29T11:37:40.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.531+0000 7f3450abd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:40.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.531+0000 7f3450abd640 1 -- 172.21.15.100:0/3569316384 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f344c0758b0 con 0x7f3444006e30 2024-03-29T11:37:40.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.531+0000 7f3450abd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:40.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.531+0000 7f3450abd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:40.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.531+0000 7f3450abd640 10 monclient: dump: 2024-03-29T11:37:40.534 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:40.534 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:40.534 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:40.535 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:40.535 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:40.535 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:40.535 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:40.535 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:40.535 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:40.535 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.531+0000 7f3452ac1640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:40.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.531+0000 7f3452ac1640 4 set_mon_vals no callback set 2024-03-29T11:37:40.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.532+0000 7f3452ac1640 10 monclient: shutdown 2024-03-29T11:37:40.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.532+0000 7f3452ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:40.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.532+0000 7f3452ac1640 1 -- 172.21.15.100:0/3569316384 >> v1:172.21.15.100:6789/0 conn(0x7f3444006e30 legacy=0x7f34440cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.532+0000 7f3452ac1640 1 -- 172.21.15.100:0/3569316384 shutdown_connections 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.532+0000 7f3452ac1640 1 -- 172.21.15.100:0/3569316384 >> 172.21.15.100:0/3569316384 conn(0x7f3444005c00 msgr2=0x7f3444006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.532+0000 7f3452ac1640 1 -- 172.21.15.100:0/3569316384 shutdown_connections 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.532+0000 7f3452ac1640 1 -- 172.21.15.100:0/3569316384 wait complete. 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.533+0000 7f3452ac1640 10 monclient: build_initial_monmap 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.533+0000 7f3452ac1640 10 monclient: monmap: 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:40.542 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.534+0000 7f3452ac1640 1 librados: starting msgr at 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.534+0000 7f3452ac1640 1 librados: starting objecter 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.534+0000 7f3452ac1640 1 Processor -- start 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.534+0000 7f3452ac1640 1 -- start start 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.534+0000 7f3452ac1640 1 librados: setting wanted keys 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.534+0000 7f3452ac1640 1 librados: calling monclient init 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.534+0000 7f3452ac1640 10 monclient: init 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3452ac1640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3452ac1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3452ac1640 10 monclient(hunting): picked mon.noname-c con 0x7f3444006e30 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3452ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f3444035e00 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3452ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f34440c9670 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3452ac1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34440326b0 con 0x7f3444035e00 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3452ac1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34440cd180 con 0x7f34440c9670 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3452ac1640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34440cd3d0 con 0x7f3444006e30 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3452ac1640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3452ac1640 10 monclient(hunting): authenticate will time out at 980.730702s 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3451abf640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f3444006e30 0x7f344402edf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:35130/0 (socket says 172.21.15.100:35130) 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f3451abf640 1 -- 172.21.15.100:0/30604032 learned_addr learned my addr 172.21.15.100:0/30604032 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f34427fc640 10 client.?.objecter ms_handle_connect 0x7f3444006e30 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f34427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f34427fc640 10 client.?.objecter ms_handle_connect 0x7f3444035e00 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.535+0000 7f34427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521214649 0 0) 0x7f34440cd3d0 con 0x7f3444006e30 2024-03-29T11:37:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3430004eb0 con 0x7f3444006e30 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375331497 0 0) 0x7f34440326b0 con 0x7f3444035e00 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34440cd3d0 con 0x7f3444035e00 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1432972140 0 0) 0x7f3430004eb0 con 0x7f3444006e30 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34440326b0 con 0x7f3444006e30 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f344c04f400 con 0x7f3444006e30 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:40.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): dump: 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2118466362 0 0) 0x7f34440cd3d0 con 0x7f3444035e00 2024-03-29T11:37:40.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3430004eb0 con 0x7f3444035e00 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3448001870 con 0x7f3444035e00 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 10 monclient(hunting): dump: 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:40.547 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2010977962 0 0) 0x7f34440326b0 con 0x7f3444006e30 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.536+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 >> v1:172.21.15.191:6789/0 conn(0x7f34440c9670 legacy=0x7f3444185460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 >> v1:172.21.15.100:6789/0 conn(0x7f3444035e00 legacy=0x7f34440bccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f34427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f34427fc640 1 monclient: found mon.c 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f34427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f34427fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34440ba510 con 0x7f3444006e30 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f34427fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f34427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f34427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f3452ac1640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f3452ac1640 10 monclient: _renew_subs 2024-03-29T11:37:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f3452ac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f3452ac1640 1 -- 172.21.15.100:0/30604032 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344402e170 con 0x7f3444006e30 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f3452ac1640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f3452ac1640 10 monclient: _renew_subs 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f3452ac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f3452ac1640 1 -- 172.21.15.100:0/30604032 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f344402f6e0 con 0x7f3444006e30 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.539+0000 7f3452ac1640 1 librados: init done 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.540+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f344c04fdd0 con 0x7f3444006e30 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.540+0000 7f34427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.540+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f344c067bf0 con 0x7f3444006e30 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.540+0000 7f34427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.540+0000 7f34427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.540+0000 7f34427fc640 10 monclient: dump: 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:40.549 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.540+0000 7f3443fff640 4 set_mon_vals no callback set 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.541+0000 7f3452ac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.542+0000 7f3452ac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.542+0000 7f3452ac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:37:40.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.542+0000 7f3452ac1640 1 -- 172.21.15.100:0/30604032 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3434005080 con 0x7f3444006e30 2024-03-29T11:37:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.543+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f344c067d50 con 0x7f3444006e30 2024-03-29T11:37:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.545+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f344c0ff6b0 con 0x7f3444006e30 2024-03-29T11:37:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.545+0000 7f34427fc640 10 client.4274.objecter ms_dispatch 0x7f344c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.545+0000 7f34427fc640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.545+0000 7f34427fc640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.545+0000 7f34427fc640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T11:37:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.546+0000 7f34512be640 10 monclient: get_auth_request con 0x7f343006dc50 auth_method 0 2024-03-29T11:37:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.546+0000 7f34427fc640 10 client.4274.objecter ms_handle_connect 0x7f343006dc50 2024-03-29T11:37:40.553 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.553+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f344c0caf90 con 0x7f3444006e30 2024-03-29T11:37:40.553 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.553+0000 7f34427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:40.553 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.553+0000 7f34427fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:40.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.950+0000 7f3452ac1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:40.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.950+0000 7f3452ac1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:40.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.950+0000 7f3452ac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:37:40.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.950+0000 7f3452ac1640 1 -- 172.21.15.100:0/30604032 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3434005330 con 0x7f3444006e30 2024-03-29T11:37:40.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.952+0000 7f34427fc640 1 -- 172.21.15.100:0/30604032 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f344c0ccec0 con 0x7f3444006e30 2024-03-29T11:37:40.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.952+0000 7f34427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:40.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.952+0000 7f34427fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:40.954 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:37:40.954 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:40.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.957+0000 7f3452ac1640 10 librados: watch_flush enter 2024-03-29T11:37:40.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.957+0000 7f3452ac1640 10 librados: watch_flush exit 2024-03-29T11:37:40.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.957+0000 7f3452ac1640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T11:37:40.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.957+0000 7f3452ac1640 10 client.4274.objecter successfully canceled tick 2024-03-29T11:37:40.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.957+0000 7f3452ac1640 1 -- 172.21.15.100:0/30604032 >> v1:172.21.15.100:6816/3220289657 conn(0x7f343006dc50 legacy=0x7f3430070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:40.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.957+0000 7f3452ac1640 10 monclient: shutdown 2024-03-29T11:37:40.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.957+0000 7f3452ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:40.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.957+0000 7f3452ac1640 1 -- 172.21.15.100:0/30604032 >> v1:172.21.15.100:6790/0 conn(0x7f3444006e30 legacy=0x7f344402edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:40.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.957+0000 7f3452ac1640 1 -- 172.21.15.100:0/30604032 shutdown_connections 2024-03-29T11:37:40.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.958+0000 7f3452ac1640 1 -- 172.21.15.100:0/30604032 >> 172.21.15.100:0/30604032 conn(0x7f3444005c00 msgr2=0x7f34440c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:40.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.958+0000 7f3452ac1640 1 -- 172.21.15.100:0/30604032 shutdown_connections 2024-03-29T11:37:40.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.958+0000 7f3452ac1640 1 -- 172.21.15.100:0/30604032 wait complete. 2024-03-29T11:37:40.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:40.958+0000 7f3452ac1640 1 librados: shutdown 2024-03-29T11:37:40.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:47.781 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:47.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.918+0000 7f3c5f10e640 10 monclient: get_monmap_and_config 2024-03-29T11:37:47.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.918+0000 7f3c5f10e640 10 monclient: build_initial_monmap 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.918+0000 7f3c5f10e640 10 monclient: monmap: 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:37:47.919623+0000 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:37:47.919623+0000 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:37:47.919 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:47.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.919+0000 7f3c5f10e640 1 Processor -- start 2024-03-29T11:37:47.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.919+0000 7f3c5f10e640 1 -- start start 2024-03-29T11:37:47.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.919+0000 7f3c5f10e640 10 monclient: init 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5f10e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5f10e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5f10e640 10 monclient(hunting): picked mon.noname-c con 0x7f3c581457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5f10e640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5813f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5f10e640 10 monclient(hunting): picked mon.noname-b con 0x7f3c580a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5f10e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c580aaef0 con 0x7f3c5813f650 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5f10e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c58145bd0 con 0x7f3c581457f0 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5f10e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c58145eb0 con 0x7f3c580a9d70 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5f10e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5f10e640 10 monclient(hunting): authenticate will time out at 988.115793s 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5d6ab640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f3c580a9d70 0x7f3c5814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:49404/0 (socket says 172.21.15.100:49404) 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.920+0000 7f3c5d6ab640 1 -- 172.21.15.100:0/2688250398 learned_addr learned my addr 172.21.15.100:0/2688250398 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184809781 0 0) 0x7f3c58145eb0 con 0x7f3c580a9d70 2024-03-29T11:37:47.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c40003520 con 0x7f3c580a9d70 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198767281 0 0) 0x7f3c58145bd0 con 0x7f3c581457f0 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c58145eb0 con 0x7f3c581457f0 2024-03-29T11:37:47.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2358558135 0 0) 0x7f3c40003520 con 0x7f3c580a9d70 2024-03-29T11:37:47.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 >> v1:172.21.15.191:6789/0 conn(0x7f3c581457f0 legacy=0x7f3c5813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:47.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.921+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 >> v1:172.21.15.100:6789/0 conn(0x7f3c5813f650 legacy=0x7f3c580b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:47.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:47.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 1 monclient: found mon.noname-b 2024-03-29T11:37:47.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:47.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:37:47.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c58146340 con 0x7f3c580a9d70 2024-03-29T11:37:47.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:47.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:47.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:47.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3c48001690 con 0x7f3c580a9d70 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 10 monclient: dump: 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:47.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c5f10e640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c5f10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c480018b0 con 0x7f3c580a9d70 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c5f10e640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.923+0000 7f3c5f10e640 4 set_mon_vals no callback set 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c577fe640 1 -- 172.21.15.100:0/2688250398 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3c48001c40 con 0x7f3c580a9d70 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c577fe640 10 monclient: dump: 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:47.928 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c5f10e640 10 monclient: shutdown 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c5f10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c5f10e640 1 -- 172.21.15.100:0/2688250398 >> v1:172.21.15.100:6790/0 conn(0x7f3c580a9d70 legacy=0x7f3c5814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c5f10e640 1 -- 172.21.15.100:0/2688250398 shutdown_connections 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c5f10e640 1 -- 172.21.15.100:0/2688250398 >> 172.21.15.100:0/2688250398 conn(0x7f3c58066a00 msgr2=0x7f3c58066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c5f10e640 1 -- 172.21.15.100:0/2688250398 shutdown_connections 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.924+0000 7f3c5f10e640 1 -- 172.21.15.100:0/2688250398 wait complete. 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.925+0000 7f3c5f10e640 10 monclient: build_initial_monmap 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.925+0000 7f3c5f10e640 10 monclient: monmap: 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:47.929 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:47.930 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:37:47.930 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:37:47.930 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:47.930 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:47.930 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:47.930 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:37:47.930 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:37:47.930 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:47.930 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.926+0000 7f3c5f10e640 1 librados: starting msgr at 2024-03-29T11:37:47.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.926+0000 7f3c5f10e640 1 librados: starting objecter 2024-03-29T11:37:47.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.926+0000 7f3c5f10e640 1 Processor -- start 2024-03-29T11:37:47.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.926+0000 7f3c5f10e640 1 -- start start 2024-03-29T11:37:47.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.926+0000 7f3c5f10e640 1 librados: setting wanted keys 2024-03-29T11:37:47.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.926+0000 7f3c5f10e640 1 librados: calling monclient init 2024-03-29T11:37:47.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.926+0000 7f3c5f10e640 10 monclient: init 2024-03-29T11:37:47.933 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.927+0000 7f3c5f10e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:47.933 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.927+0000 7f3c5f10e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:47.933 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.927+0000 7f3c5f10e640 10 monclient(hunting): picked mon.noname-b con 0x7f3c580a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.927+0000 7f3c5f10e640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5813f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.927+0000 7f3c5ceaa640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3c580a9d70 0x7f3c58142810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:56140/0 (socket says 172.21.15.100:56140) 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.928+0000 7f3c57fff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f3c5813f650 0x7f3c58140420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:50402/0 (socket says 172.21.15.100:50402) 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.929+0000 7f3c5ceaa640 1 -- 172.21.15.100:0/1835884506 learned_addr learned my addr 172.21.15.100:0/1835884506 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.929+0000 7f3c5f10e640 10 monclient(hunting): picked mon.noname-a con 0x7f3c581457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.929+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c58141850 con 0x7f3c581457f0 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.929+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c58141da0 con 0x7f3c580a9d70 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c58142080 con 0x7f3c5813f650 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c5f10e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c5f10e640 10 monclient(hunting): authenticate will time out at 988.125111s 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 client.?.objecter ms_handle_connect 0x7f3c580a9d70 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:47.934 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 client.?.objecter ms_handle_connect 0x7f3c5813f650 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 client.?.objecter ms_handle_connect 0x7f3c581457f0 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528213918 0 0) 0x7f3c58141da0 con 0x7f3c580a9d70 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c44004eb0 con 0x7f3c580a9d70 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193736413 0 0) 0x7f3c58141850 con 0x7f3c581457f0 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:47.935 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c58141da0 con 0x7f3c581457f0 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769613396 0 0) 0x7f3c58142080 con 0x7f3c5813f650 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c58141850 con 0x7f3c5813f650 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2654712680 0 0) 0x7f3c44004eb0 con 0x7f3c580a9d70 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c58142080 con 0x7f3c580a9d70 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3c50001da0 con 0x7f3c580a9d70 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): dump: 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:47.936 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2094377949 0 0) 0x7f3c58141850 con 0x7f3c5813f650 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c44004eb0 con 0x7f3c5813f650 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3c4c001690 con 0x7f3c5813f650 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:47.937 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.930+0000 7f3c557fa640 10 monclient(hunting): dump: 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4174159255 0 0) 0x7f3c58141da0 con 0x7f3c581457f0 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c58141850 con 0x7f3c581457f0 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3c48001430 con 0x7f3c581457f0 2024-03-29T11:37:47.938 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 10 monclient(hunting): dump: 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2821047693 0 0) 0x7f3c58142080 con 0x7f3c580a9d70 2024-03-29T11:37:47.939 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 >> v1:172.21.15.100:6790/0 conn(0x7f3c5813f650 legacy=0x7f3c58140420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 >> v1:172.21.15.100:6789/0 conn(0x7f3c581457f0 legacy=0x7f3c58134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 1 monclient: found mon.b 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c58141ac0 con 0x7f3c580a9d70 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c5f10e640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c5f10e640 10 monclient: _renew_subs 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c5f10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c58138f60 con 0x7f3c580a9d70 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c5f10e640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:47.940 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c5f10e640 10 monclient: _renew_subs 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c5f10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c58140d10 con 0x7f3c580a9d70 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c5f10e640 1 librados: init done 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c50002080 con 0x7f3c580a9d70 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.931+0000 7f3c557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.932+0000 7f3c56ffd640 4 set_mon_vals no callback set 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.932+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3c50003de0 con 0x7f3c580a9d70 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.932+0000 7f3c557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.932+0000 7f3c557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.932+0000 7f3c557fa640 10 monclient: dump: 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:47.941 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:47.942 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:47.942 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:47.942 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:47.942 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:47.942 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:47.942 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.933+0000 7f3c5f10e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:47.942 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.933+0000 7f3c5f10e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:47.942 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.933+0000 7f3c5f10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:47.942 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.933+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c38005080 con 0x7f3c580a9d70 2024-03-29T11:37:47.943 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.935+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f3c50003fc0 con 0x7f3c580a9d70 2024-03-29T11:37:47.943 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.936+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f3c50084eb0 con 0x7f3c580a9d70 2024-03-29T11:37:47.943 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.936+0000 7f3c557fa640 10 client.4285.objecter ms_dispatch 0x7f3c58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:47.943 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.936+0000 7f3c557fa640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:47.943 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.936+0000 7f3c557fa640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:47.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.936+0000 7f3c557fa640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-29T11:37:47.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.936+0000 7f3c57fff640 10 monclient: get_auth_request con 0x7f3c4406e1b0 auth_method 0 2024-03-29T11:37:47.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.936+0000 7f3c557fa640 10 client.4285.objecter ms_handle_connect 0x7f3c4406e1b0 2024-03-29T11:37:47.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.943+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c50050790 con 0x7f3c580a9d70 2024-03-29T11:37:47.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.943+0000 7f3c557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:47.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:47.943+0000 7f3c557fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:48.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.250+0000 7f3c5f10e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:48.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.250+0000 7f3c5f10e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:48.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.250+0000 7f3c5f10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:48.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.250+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c38005330 con 0x7f3c580a9d70 2024-03-29T11:37:48.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.251+0000 7f3c557fa640 1 -- 172.21.15.100:0/1835884506 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f3c500526c0 con 0x7f3c580a9d70 2024-03-29T11:37:48.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.251+0000 7f3c557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:48.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.251+0000 7f3c557fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:48.251 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:37:48.251 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:48.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.254+0000 7f3c5f10e640 10 librados: watch_flush enter 2024-03-29T11:37:48.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.254+0000 7f3c5f10e640 10 librados: watch_flush exit 2024-03-29T11:37:48.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.254+0000 7f3c5f10e640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-29T11:37:48.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.254+0000 7f3c5f10e640 10 client.4285.objecter successfully canceled tick 2024-03-29T11:37:48.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.254+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 >> v1:172.21.15.100:6816/3220289657 conn(0x7f3c4406e1b0 legacy=0x7f3c44070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:48.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.254+0000 7f3c5f10e640 10 monclient: shutdown 2024-03-29T11:37:48.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.254+0000 7f3c5f10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:48.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.254+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 >> v1:172.21.15.191:6789/0 conn(0x7f3c580a9d70 legacy=0x7f3c58142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:48.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.255+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 shutdown_connections 2024-03-29T11:37:48.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.255+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 >> 172.21.15.100:0/1835884506 conn(0x7f3c58066a00 msgr2=0x7f3c580b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:48.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.255+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 shutdown_connections 2024-03-29T11:37:48.256 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.255+0000 7f3c5f10e640 1 -- 172.21.15.100:0/1835884506 wait complete. 2024-03-29T11:37:48.256 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:48.256+0000 7f3c5f10e640 1 librados: shutdown 2024-03-29T11:37:48.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:55.076 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:55.221 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.220+0000 7f4c23435640 10 monclient: get_monmap_and_config 2024-03-29T11:37:55.221 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.220+0000 7f4c23435640 10 monclient: build_initial_monmap 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.220+0000 7f4c23435640 10 monclient: monmap: 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:37:55.221374+0000 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:37:55.221374+0000 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.221+0000 7f4c23435640 1 Processor -- start 2024-03-29T11:37:55.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.221+0000 7f4c23435640 1 -- start start 2024-03-29T11:37:55.223 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.221+0000 7f4c23435640 10 monclient: init 2024-03-29T11:37:55.223 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.222+0000 7f4c23435640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.222+0000 7f4c23435640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.222+0000 7f4c23435640 10 monclient(hunting): picked mon.noname-c con 0x7f4c14034870 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.222+0000 7f4c23435640 10 monclient(hunting): picked mon.noname-b con 0x7f4c1402a660 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.222+0000 7f4c23435640 10 monclient(hunting): picked mon.noname-a con 0x7f4c140327f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.222+0000 7f4c23435640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1402c840 con 0x7f4c140327f0 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.222+0000 7f4c23435640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1402a140 con 0x7f4c14034870 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.222+0000 7f4c23435640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c140c75d0 con 0x7f4c1402a660 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.222+0000 7f4c23435640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.222+0000 7f4c23435640 10 monclient(hunting): authenticate will time out at 995.417664s 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.223+0000 7f4c211d1640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4c14034870 0x7f4c140c6ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:56888/0 (socket says 172.21.15.100:56888) 2024-03-29T11:37:55.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.223+0000 7f4c211d1640 1 -- 172.21.15.100:0/1828520615 learned_addr learned my addr 172.21.15.100:0/1828520615 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:55.228 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093600442 0 0) 0x7f4c140c75d0 con 0x7f4c1402a660 2024-03-29T11:37:55.228 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c08003520 con 0x7f4c1402a660 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226723112 0 0) 0x7f4c1402a140 con 0x7f4c14034870 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c140c75d0 con 0x7f4c14034870 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056024709 0 0) 0x7f4c1402c840 con 0x7f4c140327f0 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:55.229 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:55.230 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:37:55.230 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.228+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c1402a140 con 0x7f4c140327f0 2024-03-29T11:37:55.230 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 816279393 0 0) 0x7f4c1402a140 con 0x7f4c140327f0 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 >> v1:172.21.15.100:6790/0 conn(0x7f4c1402a660 legacy=0x7f4c140320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 >> v1:172.21.15.191:6789/0 conn(0x7f4c14034870 legacy=0x7f4c140c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 1 monclient: found mon.noname-a 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c14033130 con 0x7f4c140327f0 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 10 monclient: _finish_auth 0 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4c0c001630 con 0x7f4c140327f0 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:55.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.229+0000 7f4c13fff640 10 monclient: dump: 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.230+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c0c001800 con 0x7f4c140327f0 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.230+0000 7f4c13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.230+0000 7f4c13fff640 1 -- 172.21.15.100:0/1828520615 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4c0c001bd0 con 0x7f4c140327f0 2024-03-29T11:37:55.232 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.230+0000 7f4c13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.230+0000 7f4c13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.230+0000 7f4c13fff640 10 monclient: dump: 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.230+0000 7f4c23435640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.230+0000 7f4c23435640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:55.233 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.230+0000 7f4c23435640 4 set_mon_vals no callback set 2024-03-29T11:37:55.234 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.231+0000 7f4c23435640 10 monclient: shutdown 2024-03-29T11:37:55.234 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.231+0000 7f4c23435640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:55.234 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.231+0000 7f4c23435640 1 -- 172.21.15.100:0/1828520615 >> v1:172.21.15.100:6789/0 conn(0x7f4c140327f0 legacy=0x7f4c140cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:55.234 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.231+0000 7f4c23435640 1 -- 172.21.15.100:0/1828520615 shutdown_connections 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.231+0000 7f4c23435640 1 -- 172.21.15.100:0/1828520615 >> 172.21.15.100:0/1828520615 conn(0x7f4c14005910 msgr2=0x7f4c14005d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.231+0000 7f4c23435640 1 -- 172.21.15.100:0/1828520615 shutdown_connections 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.231+0000 7f4c23435640 1 -- 172.21.15.100:0/1828520615 wait complete. 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.232+0000 7f4c23435640 10 monclient: build_initial_monmap 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.232+0000 7f4c23435640 10 monclient: monmap: 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:37:55.235 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.232+0000 7f4c23435640 1 librados: starting msgr at 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.232+0000 7f4c23435640 1 librados: starting objecter 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.232+0000 7f4c23435640 1 Processor -- start 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.232+0000 7f4c23435640 1 -- start start 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.232+0000 7f4c23435640 1 librados: setting wanted keys 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.232+0000 7f4c23435640 1 librados: calling monclient init 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.232+0000 7f4c23435640 10 monclient: init 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c23435640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c23435640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c23435640 10 monclient(hunting): picked mon.noname-b con 0x7f4c1402a660 addr v1:172.21.15.191:6789/0 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c23435640 10 monclient(hunting): picked mon.noname-a con 0x7f4c140327f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c23435640 10 monclient(hunting): picked mon.noname-c con 0x7f4c14034870 addr v1:172.21.15.100:6790/0 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c23435640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c140c46c0 con 0x7f4c140327f0 2024-03-29T11:37:55.236 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c23435640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c140c4930 con 0x7f4c1402a660 2024-03-29T11:37:55.237 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c23435640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c140c3b50 con 0x7f4c14034870 2024-03-29T11:37:55.237 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c23435640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:55.237 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c23435640 10 monclient(hunting): authenticate will time out at 995.428818s 2024-03-29T11:37:55.237 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c211d1640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4c1402a660 0x7f4c140c3250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:57312/0 (socket says 172.21.15.100:57312) 2024-03-29T11:37:55.237 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.233+0000 7f4c211d1640 1 -- 172.21.15.100:0/2369223091 learned_addr learned my addr 172.21.15.100:0/2369223091 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:37:55.238 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.234+0000 7f4c11ffb640 10 client.?.objecter ms_handle_connect 0x7f4c1402a660 2024-03-29T11:37:55.238 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.234+0000 7f4c11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:55.238 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.236+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693460215 0 0) 0x7f4c140c4930 con 0x7f4c1402a660 2024-03-29T11:37:55.238 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:55.238 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:55.238 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:55.238 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:37:55.238 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c00004eb0 con 0x7f4c1402a660 2024-03-29T11:37:55.238 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 10 client.?.objecter ms_handle_connect 0x7f4c140327f0 2024-03-29T11:37:55.238 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:55.239 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1996600137 0 0) 0x7f4c00004eb0 con 0x7f4c1402a660 2024-03-29T11:37:55.239 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c140c4930 con 0x7f4c1402a660 2024-03-29T11:37:55.239 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632561962 0 0) 0x7f4c140c46c0 con 0x7f4c140327f0 2024-03-29T11:37:55.239 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:55.239 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:55.239 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:55.239 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:37:55.239 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.237+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c00004eb0 con 0x7f4c140327f0 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 client.?.objecter ms_handle_connect 0x7f4c14034870 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4c18001970 con 0x7f4c1402a660 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient(hunting): dump: 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:55.243 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455444844 0 0) 0x7f4c140c3b50 con 0x7f4c14034870 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c140c46c0 con 0x7f4c14034870 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1555692610 0 0) 0x7f4c140c4930 con 0x7f4c1402a660 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 >> v1:172.21.15.100:6790/0 conn(0x7f4c14034870 legacy=0x7f4c1416b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 >> v1:172.21.15.100:6789/0 conn(0x7f4c140327f0 legacy=0x7f4c1402d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:55.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 1 monclient: found mon.b 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1402d750 con 0x7f4c1402a660 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.239+0000 7f4c11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.240+0000 7f4c23435640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.240+0000 7f4c23435640 10 monclient: _renew_subs 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.240+0000 7f4c23435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.240+0000 7f4c23435640 1 -- 172.21.15.100:0/2369223091 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c140c25b0 con 0x7f4c1402a660 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.240+0000 7f4c23435640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.240+0000 7f4c23435640 10 monclient: _renew_subs 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.240+0000 7f4c23435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:55.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.240+0000 7f4c23435640 1 -- 172.21.15.100:0/2369223091 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c140c0530 con 0x7f4c1402a660 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.240+0000 7f4c23435640 1 librados: init done 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.242+0000 7f4c23435640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.242+0000 7f4c23435640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.242+0000 7f4c23435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.242+0000 7f4c23435640 1 -- 172.21.15.100:0/2369223091 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c140c3960 con 0x7f4c1402a660 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.242+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c18001b40 con 0x7f4c1402a660 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.242+0000 7f4c11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.242+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4c18004a90 con 0x7f4c1402a660 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.242+0000 7f4c11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.242+0000 7f4c11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.242+0000 7f4c11ffb640 10 monclient: dump: 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:37:55.246 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:55.247 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:37:55.247 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:55.247 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:37:55.247 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:37:55.247 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:37:55.247 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:37:55.247 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:37:55.248 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.244+0000 7f4c137fe640 4 set_mon_vals no callback set 2024-03-29T11:37:55.248 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.245+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f4c1801aa60 con 0x7f4c1402a660 2024-03-29T11:37:55.248 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.246+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f4c18084a60 con 0x7f4c1402a660 2024-03-29T11:37:55.248 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.246+0000 7f4c11ffb640 10 client.4291.objecter ms_dispatch 0x7f4c1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:55.248 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.246+0000 7f4c11ffb640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:55.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.246+0000 7f4c11ffb640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:55.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.246+0000 7f4c11ffb640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T11:37:55.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.246+0000 7f4c209d0640 10 monclient: get_auth_request con 0x7f4c0006ddb0 auth_method 0 2024-03-29T11:37:55.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.247+0000 7f4c11ffb640 10 client.4291.objecter ms_handle_connect 0x7f4c0006ddb0 2024-03-29T11:37:55.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.249+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c18050340 con 0x7f4c1402a660 2024-03-29T11:37:55.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.249+0000 7f4c11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:55.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.249+0000 7f4c11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:55.582 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.581+0000 7f4c23435640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:55.582 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.581+0000 7f4c23435640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:55.582 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.581+0000 7f4c23435640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:37:55.582 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.581+0000 7f4c23435640 1 -- 172.21.15.100:0/2369223091 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c14004cf0 con 0x7f4c1402a660 2024-03-29T11:37:55.583 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.583+0000 7f4c11ffb640 1 -- 172.21.15.100:0/2369223091 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f4c18052270 con 0x7f4c1402a660 2024-03-29T11:37:55.583 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.583+0000 7f4c11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:55.583 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.583+0000 7f4c11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:55.585 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:37:55.585 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:55.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.588+0000 7f4c23435640 10 librados: watch_flush enter 2024-03-29T11:37:55.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.588+0000 7f4c23435640 10 librados: watch_flush exit 2024-03-29T11:37:55.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.588+0000 7f4c23435640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T11:37:55.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.588+0000 7f4c23435640 10 client.4291.objecter successfully canceled tick 2024-03-29T11:37:55.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.588+0000 7f4c23435640 1 -- 172.21.15.100:0/2369223091 >> v1:172.21.15.100:6816/3220289657 conn(0x7f4c0006ddb0 legacy=0x7f4c00070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:55.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.588+0000 7f4c23435640 10 monclient: shutdown 2024-03-29T11:37:55.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.588+0000 7f4c23435640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:55.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.588+0000 7f4c23435640 1 -- 172.21.15.100:0/2369223091 >> v1:172.21.15.191:6789/0 conn(0x7f4c1402a660 legacy=0x7f4c140c3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:55.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.589+0000 7f4c23435640 1 -- 172.21.15.100:0/2369223091 shutdown_connections 2024-03-29T11:37:55.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.589+0000 7f4c23435640 1 -- 172.21.15.100:0/2369223091 >> 172.21.15.100:0/2369223091 conn(0x7f4c14005910 msgr2=0x7f4c140043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:55.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.590+0000 7f4c23435640 1 -- 172.21.15.100:0/2369223091 shutdown_connections 2024-03-29T11:37:55.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.590+0000 7f4c23435640 1 -- 172.21.15.100:0/2369223091 wait complete. 2024-03-29T11:37:55.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:37:55.590+0000 7f4c23435640 1 librados: shutdown 2024-03-29T11:37:55.608 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:02.415 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:02.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.561+0000 7fb2e0ee0640 10 monclient: get_monmap_and_config 2024-03-29T11:38:02.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.561+0000 7fb2e0ee0640 10 monclient: build_initial_monmap 2024-03-29T11:38:02.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.561+0000 7fb2e0ee0640 10 monclient: monmap: 2024-03-29T11:38:02.563 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:02.563 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:02.563 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:38:02.562597+0000 2024-03-29T11:38:02.564 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:38:02.562597+0000 2024-03-29T11:38:02.564 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:02.564 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:02.564 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:02.564 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:38:02.564 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:38:02.564 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:02.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.562+0000 7fb2e0ee0640 1 Processor -- start 2024-03-29T11:38:02.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.562+0000 7fb2e0ee0640 1 -- start start 2024-03-29T11:38:02.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.562+0000 7fb2e0ee0640 10 monclient: init 2024-03-29T11:38:02.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.563+0000 7fb2e0ee0640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:02.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.563+0000 7fb2e0ee0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:02.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.563+0000 7fb2e0ee0640 10 monclient(hunting): picked mon.noname-a con 0x7fb2dc0af6c0 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:02.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.563+0000 7fb2e0ee0640 10 monclient(hunting): picked mon.noname-b con 0x7fb2dc14f6b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:02.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.563+0000 7fb2e0ee0640 10 monclient(hunting): picked mon.noname-c con 0x7fb2dc152980 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:02.568 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.563+0000 7fb2e0ee0640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0a94e0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.568 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.563+0000 7fb2e0ee0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0afb60 con 0x7fb2dc152980 2024-03-29T11:38:02.568 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.563+0000 7fb2e0ee0640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0aab80 con 0x7fb2dc14f6b0 2024-03-29T11:38:02.568 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.563+0000 7fb2e0ee0640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:02.568 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.563+0000 7fb2e0ee0640 10 monclient(hunting): authenticate will time out at 1002.758877s 2024-03-29T11:38:02.568 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.567+0000 7fb2da59c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fb2dc0af6c0 0x7fb2dc0ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:38637/0 (socket says 172.21.15.100:38637) 2024-03-29T11:38:02.568 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.567+0000 7fb2da59c640 1 -- 172.21.15.100:0/1982251664 learned_addr learned my addr 172.21.15.100:0/1982251664 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:02.568 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.567+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413558789 0 0) 0x7fb2dc0a94e0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.569 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.567+0000 7fb2d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:02.569 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.567+0000 7fb2d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:02.569 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.567+0000 7fb2d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:02.572 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.567+0000 7fb2d959a640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:38:02.572 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.567+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c4003520 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362481000 0 0) 0x7fb2dc0afb60 con 0x7fb2dc152980 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2dc0a94e0 con 0x7fb2dc152980 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1669266473 0 0) 0x7fb2c4003520 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 >> v1:172.21.15.100:6790/0 conn(0x7fb2dc14f6b0 legacy=0x7fb2dc14fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 >> v1:172.21.15.191:6789/0 conn(0x7fb2dc152980 legacy=0x7fb2dc152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 1 monclient: found mon.noname-a 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0aa680 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.573 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient: _finish_auth 0 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb2cc001a10 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2d959a640 10 monclient: dump: 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:02.574 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2e0ee0640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.568+0000 7fb2e0ee0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.570+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2cc001b70 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.570+0000 7fb2d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.570+0000 7fb2d959a640 1 -- 172.21.15.100:0/1982251664 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb2cc001fc0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.570+0000 7fb2d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.570+0000 7fb2d959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.570+0000 7fb2d959a640 10 monclient: dump: 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:02.575 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.570+0000 7fb2e0ee0640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.570+0000 7fb2e0ee0640 4 set_mon_vals no callback set 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.571+0000 7fb2e0ee0640 10 monclient: shutdown 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.571+0000 7fb2e0ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.571+0000 7fb2e0ee0640 1 -- 172.21.15.100:0/1982251664 >> v1:172.21.15.100:6789/0 conn(0x7fb2dc0af6c0 legacy=0x7fb2dc0ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.572+0000 7fb2e0ee0640 1 -- 172.21.15.100:0/1982251664 shutdown_connections 2024-03-29T11:38:02.576 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.572+0000 7fb2e0ee0640 1 -- 172.21.15.100:0/1982251664 >> 172.21.15.100:0/1982251664 conn(0x7fb2dc0b35e0 msgr2=0x7fb2dc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:02.577 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.574+0000 7fb2e0ee0640 1 -- 172.21.15.100:0/1982251664 shutdown_connections 2024-03-29T11:38:02.577 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.574+0000 7fb2e0ee0640 1 -- 172.21.15.100:0/1982251664 wait complete. 2024-03-29T11:38:02.577 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.575+0000 7fb2e0ee0640 10 monclient: build_initial_monmap 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.575+0000 7fb2e0ee0640 10 monclient: monmap: 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:02.580 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.576+0000 7fb2e0ee0640 1 librados: starting msgr at 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.576+0000 7fb2e0ee0640 1 librados: starting objecter 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.576+0000 7fb2e0ee0640 1 Processor -- start 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.576+0000 7fb2e0ee0640 1 -- start start 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.576+0000 7fb2e0ee0640 1 librados: setting wanted keys 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.576+0000 7fb2e0ee0640 1 librados: calling monclient init 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.576+0000 7fb2e0ee0640 10 monclient: init 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.577+0000 7fb2e0ee0640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.577+0000 7fb2e0ee0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.577+0000 7fb2e0ee0640 10 monclient(hunting): picked mon.noname-b con 0x7fb2dc0af6c0 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.579+0000 7fb2e0ee0640 10 monclient(hunting): picked mon.noname-c con 0x7fb2dc14f6b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.579+0000 7fb2e0ee0640 10 monclient(hunting): picked mon.noname-a con 0x7fb2dc152980 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.579+0000 7fb2e0ee0640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc153d60 con 0x7fb2dc152980 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.579+0000 7fb2e0ee0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc1542b0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.581 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.579+0000 7fb2e0ee0640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc153fd0 con 0x7fb2dc14f6b0 2024-03-29T11:38:02.582 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.579+0000 7fb2e0ee0640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:02.582 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.579+0000 7fb2e0ee0640 10 monclient(hunting): authenticate will time out at 1002.774334s 2024-03-29T11:38:02.582 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.579+0000 7fb2da59c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fb2dc0af6c0 0x7fb2dc1f8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:52054/0 (socket says 172.21.15.100:52054) 2024-03-29T11:38:02.582 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.579+0000 7fb2da59c640 1 -- 172.21.15.100:0/2011552670 learned_addr learned my addr 172.21.15.100:0/2011552670 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:02.585 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.580+0000 7fb2caffd640 10 client.?.objecter ms_handle_connect 0x7fb2dc0af6c0 2024-03-29T11:38:02.585 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.580+0000 7fb2caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:02.585 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.580+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947492944 0 0) 0x7fb2dc1542b0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.580+0000 7fb2caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.580+0000 7fb2caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.580+0000 7fb2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.580+0000 7fb2caffd640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2bc004eb0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3808523207 0 0) 0x7fb2bc004eb0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2dc1542b0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb2cc001e80 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 10 monclient(hunting): dump: 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:02.586 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3466833939 0 0) 0x7fb2dc1542b0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 >> v1:172.21.15.100:6790/0 conn(0x7fb2dc14f6b0 legacy=0x7fb2dc1507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.581+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 >> v1:172.21.15.100:6789/0 conn(0x7fb2dc152980 legacy=0x7fb2dc150f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2caffd640 1 monclient: found mon.b 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:02.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2caffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc154590 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2caffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2e0ee0640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2e0ee0640 10 monclient: _renew_subs 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2e0ee0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2e0ee0640 1 -- 172.21.15.100:0/2011552670 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc2008f0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2e0ee0640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2e0ee0640 10 monclient: _renew_subs 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2e0ee0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2e0ee0640 1 -- 172.21.15.100:0/2011552670 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc200c60 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2e0ee0640 1 librados: init done 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2cc002050 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:02.588 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.583+0000 7fb2d8d99640 4 set_mon_vals no callback set 2024-03-29T11:38:02.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.584+0000 7fb2c8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:02.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.584+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb2cc005190 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.584+0000 7fb2caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:02.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.584+0000 7fb2caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:02.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.584+0000 7fb2caffd640 10 monclient: dump: 2024-03-29T11:38:02.589 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:02.589 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.584+0000 7fb2c8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.584+0000 7fb2c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.584+0000 7fb2c8ff9640 1 -- 172.21.15.100:0/2011552670 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b4005080 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.585+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fb2cc01b1e0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.586+0000 7fb2d9d9b640 10 monclient: get_auth_request con 0x7fb2bc06dbc0 auth_method 0 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.587+0000 7fb2caffd640 10 client.4297.objecter ms_handle_connect 0x7fb2bc06dbc0 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.587+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fb2cc084fd0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.587+0000 7fb2caffd640 10 client.4297.objecter ms_dispatch 0x7fb2dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:02.591 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.587+0000 7fb2caffd640 3 client.4297.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:02.591 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.587+0000 7fb2caffd640 3 client.4297.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:02.591 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.587+0000 7fb2caffd640 20 client.4297.objecter dump_active .. 0 homeless 2024-03-29T11:38:02.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.593+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2cc0508b0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.593+0000 7fb2caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:02.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.593+0000 7fb2caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:02.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.918+0000 7fb2c8ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:02.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.918+0000 7fb2c8ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:02.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.918+0000 7fb2c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:38:02.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.918+0000 7fb2c8ff9640 1 -- 172.21.15.100:0/2011552670 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2b4005330 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.919+0000 7fb2caffd640 1 -- 172.21.15.100:0/2011552670 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fb2cc0527e0 con 0x7fb2dc0af6c0 2024-03-29T11:38:02.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.919+0000 7fb2caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:02.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.919+0000 7fb2caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:02.921 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:38:02.921 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:02.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.923+0000 7fb2c8ff9640 10 librados: watch_flush enter 2024-03-29T11:38:02.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.923+0000 7fb2c8ff9640 10 librados: watch_flush exit 2024-03-29T11:38:02.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.923+0000 7fb2c8ff9640 20 client.4297.objecter shutdown clearing up homeless session... 2024-03-29T11:38:02.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.923+0000 7fb2c8ff9640 10 client.4297.objecter successfully canceled tick 2024-03-29T11:38:02.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.923+0000 7fb2c8ff9640 1 -- 172.21.15.100:0/2011552670 >> v1:172.21.15.100:6816/3220289657 conn(0x7fb2bc06dbc0 legacy=0x7fb2bc070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:02.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.923+0000 7fb2c8ff9640 10 monclient: shutdown 2024-03-29T11:38:02.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.923+0000 7fb2c8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:02.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.923+0000 7fb2c8ff9640 1 -- 172.21.15.100:0/2011552670 >> v1:172.21.15.191:6789/0 conn(0x7fb2dc0af6c0 legacy=0x7fb2dc1f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:02.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.924+0000 7fb2c8ff9640 1 -- 172.21.15.100:0/2011552670 shutdown_connections 2024-03-29T11:38:02.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.924+0000 7fb2c8ff9640 1 -- 172.21.15.100:0/2011552670 >> 172.21.15.100:0/2011552670 conn(0x7fb2dc0b35e0 msgr2=0x7fb2dc0ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:02.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.925+0000 7fb2c8ff9640 1 -- 172.21.15.100:0/2011552670 shutdown_connections 2024-03-29T11:38:02.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.925+0000 7fb2c8ff9640 1 -- 172.21.15.100:0/2011552670 wait complete. 2024-03-29T11:38:02.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:02.925+0000 7fb2c8ff9640 1 librados: shutdown 2024-03-29T11:38:02.943 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:09.749 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.896+0000 7f04d5ed6640 10 monclient: get_monmap_and_config 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.896+0000 7f04d5ed6640 10 monclient: build_initial_monmap 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.896+0000 7f04d5ed6640 10 monclient: monmap: 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:38:09.897779+0000 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:38:09.897779+0000 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:09.897 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:38:09.898 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:38:09.898 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.897+0000 7f04d5ed6640 1 Processor -- start 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.897+0000 7f04d5ed6640 1 -- start start 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.897+0000 7f04d5ed6640 10 monclient: init 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.898+0000 7f04d5ed6640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.898+0000 7f04d5ed6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.898+0000 7f04d5ed6640 10 monclient(hunting): picked mon.noname-b con 0x7f04d00b4330 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.898+0000 7f04d5ed6640 10 monclient(hunting): picked mon.noname-c con 0x7f04d01470b0 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.898+0000 7f04d5ed6640 10 monclient(hunting): picked mon.noname-a con 0x7f04d01439b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.898+0000 7f04d5ed6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d006f2a0 con 0x7f04d01439b0 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.898+0000 7f04d5ed6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d0146b90 con 0x7f04d01470b0 2024-03-29T11:38:09.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.898+0000 7f04d5ed6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00b2ca0 con 0x7f04d00b4330 2024-03-29T11:38:09.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.898+0000 7f04d5ed6640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:09.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.898+0000 7f04d5ed6640 10 monclient(hunting): authenticate will time out at 1010.093884s 2024-03-29T11:38:09.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.902+0000 7f04cffff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f04d01439b0 0x7f04d0151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:44569/0 (socket says 172.21.15.100:44569) 2024-03-29T11:38:09.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.902+0000 7f04cffff640 1 -- 172.21.15.100:0/1546975285 learned_addr learned my addr 172.21.15.100:0/1546975285 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:09.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.901+0000 7f04ceffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f04d01470b0 0x7f04d00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:46726/0 (socket says 172.21.15.100:46726) 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.902+0000 7f04ce7fc640 1 -- 172.21.15.100:0/1546975285 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935083639 0 0) 0x7f04d006f2a0 con 0x7f04d01439b0 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.902+0000 7f04ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.902+0000 7f04ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.902+0000 7f04ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.902+0000 7f04ce7fc640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.902+0000 7f04ce7fc640 1 -- 172.21.15.100:0/1546975285 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b8003520 con 0x7f04d01439b0 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 1 -- 172.21.15.100:0/1546975285 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1792320303 0 0) 0x7f04b8003520 con 0x7f04d01439b0 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 1 -- 172.21.15.100:0/1546975285 >> v1:172.21.15.100:6790/0 conn(0x7f04d00b4330 legacy=0x7f04d00b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 1 -- 172.21.15.100:0/1546975285 >> v1:172.21.15.191:6789/0 conn(0x7f04d01470b0 legacy=0x7f04d00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 1 monclient: found mon.noname-a 2024-03-29T11:38:09.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:09.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:38:09.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 1 -- 172.21.15.100:0/1546975285 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d006e0c0 con 0x7f04d01439b0 2024-03-29T11:38:09.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:09.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:09.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:09.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 1 -- 172.21.15.100:0/1546975285 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f04c8001a10 con 0x7f04d01439b0 2024-03-29T11:38:09.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:09.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: dump: 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04d5ed6640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04d5ed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 1 -- 172.21.15.100:0/1546975285 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04c8001be0 con 0x7f04d01439b0 2024-03-29T11:38:09.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04d5ed6640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04d5ed6640 4 set_mon_vals no callback set 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 1 -- 172.21.15.100:0/1546975285 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f04c8002030 con 0x7f04d01439b0 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.903+0000 7f04ce7fc640 10 monclient: dump: 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:09.908 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:09.909 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:09.909 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:09.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.904+0000 7f04d5ed6640 10 monclient: shutdown 2024-03-29T11:38:09.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.904+0000 7f04d5ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:09.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.904+0000 7f04d5ed6640 1 -- 172.21.15.100:0/1546975285 >> v1:172.21.15.100:6789/0 conn(0x7f04d01439b0 legacy=0x7f04d0151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.906+0000 7f04d5ed6640 1 -- 172.21.15.100:0/1546975285 shutdown_connections 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.906+0000 7f04d5ed6640 1 -- 172.21.15.100:0/1546975285 >> 172.21.15.100:0/1546975285 conn(0x7f04d0066a00 msgr2=0x7f04d0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.907+0000 7f04d5ed6640 1 -- 172.21.15.100:0/1546975285 shutdown_connections 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.907+0000 7f04d5ed6640 1 -- 172.21.15.100:0/1546975285 wait complete. 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.908+0000 7f04d5ed6640 10 monclient: build_initial_monmap 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.908+0000 7f04d5ed6640 10 monclient: monmap: 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:09.910 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:09.911 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:38:09.911 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:38:09.911 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:09.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.909+0000 7f04d5ed6640 1 librados: starting msgr at 2024-03-29T11:38:09.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.909+0000 7f04d5ed6640 1 librados: starting objecter 2024-03-29T11:38:09.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.909+0000 7f04d5ed6640 1 Processor -- start 2024-03-29T11:38:09.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.911+0000 7f04d5ed6640 1 -- start start 2024-03-29T11:38:09.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.911+0000 7f04d5ed6640 1 librados: setting wanted keys 2024-03-29T11:38:09.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.911+0000 7f04d5ed6640 1 librados: calling monclient init 2024-03-29T11:38:09.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.911+0000 7f04d5ed6640 10 monclient: init 2024-03-29T11:38:09.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.912+0000 7f04d5ed6640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:09.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.912+0000 7f04d5ed6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:09.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.912+0000 7f04d5ed6640 10 monclient(hunting): picked mon.noname-b con 0x7f04d00b4330 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:09.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.912+0000 7f04d5ed6640 10 monclient(hunting): picked mon.noname-c con 0x7f04d01439b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:09.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.912+0000 7f04d5ed6640 10 monclient(hunting): picked mon.noname-a con 0x7f04d01470b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:09.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.912+0000 7f04d5ed6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d01d1a50 con 0x7f04d01470b0 2024-03-29T11:38:09.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.912+0000 7f04d5ed6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d011f770 con 0x7f04d00b4330 2024-03-29T11:38:09.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.912+0000 7f04d5ed6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d01df640 con 0x7f04d01439b0 2024-03-29T11:38:09.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.912+0000 7f04d5ed6640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:09.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.912+0000 7f04d5ed6640 10 monclient(hunting): authenticate will time out at 1010.107646s 2024-03-29T11:38:09.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.915+0000 7f04cffff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f04d01470b0 0x7f04d01203a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:45383/0 (socket says 172.21.15.100:45383) 2024-03-29T11:38:09.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.915+0000 7f04cffff640 1 -- 172.21.15.100:0/726965765 learned_addr learned my addr 172.21.15.100:0/726965765 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:09.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.915+0000 7f04b7fff640 10 client.?.objecter ms_handle_connect 0x7f04d01470b0 2024-03-29T11:38:09.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:09.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032486762 0 0) 0x7f04d01d1a50 con 0x7f04d01470b0 2024-03-29T11:38:09.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04bc004eb0 con 0x7f04d01470b0 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 client.?.objecter ms_handle_connect 0x7f04d00b4330 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546389008 0 0) 0x7f04d011f770 con 0x7f04d00b4330 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d01d1a50 con 0x7f04d00b4330 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.916+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1683275793 0 0) 0x7f04bc004eb0 con 0x7f04d01470b0 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.917+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04d011f770 con 0x7f04d01470b0 2024-03-29T11:38:09.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.917+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f04c8001f30 con 0x7f04d01470b0 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.917+0000 7f04b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.917+0000 7f04b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.917+0000 7f04b7fff640 10 monclient(hunting): dump: 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:09.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.917+0000 7f04b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:09.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.917+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 963371625 0 0) 0x7f04d011f770 con 0x7f04d01470b0 2024-03-29T11:38:09.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.917+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 >> v1:172.21.15.100:6790/0 conn(0x7f04d01439b0 legacy=0x7f04d011fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:09.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.917+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 >> v1:172.21.15.191:6789/0 conn(0x7f04d00b4330 legacy=0x7f04d01def30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:09.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.917+0000 7f04b7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:09.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.918+0000 7f04b7fff640 1 monclient: found mon.a 2024-03-29T11:38:09.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.918+0000 7f04b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:09.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.918+0000 7f04b7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.918+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d01df7d0 con 0x7f04d01470b0 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.918+0000 7f04b7fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.918+0000 7f04b7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.918+0000 7f04b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.920+0000 7f04d5ed6640 5 monclient: authenticate success, global_id 4305 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.920+0000 7f04d5ed6640 10 monclient: _renew_subs 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.920+0000 7f04d5ed6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.920+0000 7f04d5ed6640 1 -- 172.21.15.100:0/726965765 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d01dfa20 con 0x7f04d01470b0 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.920+0000 7f04d5ed6640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.920+0000 7f04d5ed6640 10 monclient: _renew_subs 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.920+0000 7f04d5ed6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.920+0000 7f04d5ed6640 1 -- 172.21.15.100:0/726965765 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d01e73f0 con 0x7f04d01470b0 2024-03-29T11:38:09.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.920+0000 7f04d5ed6640 1 librados: init done 2024-03-29T11:38:09.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.922+0000 7f04b5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:09.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.922+0000 7f04b5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:09.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.922+0000 7f04b5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:38:09.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.922+0000 7f04b5ffb640 1 -- 172.21.15.100:0/726965765 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d011f9b0 con 0x7f04d01470b0 2024-03-29T11:38:09.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.924+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04c8002090 con 0x7f04d01470b0 2024-03-29T11:38:09.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.924+0000 7f04b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.924+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f04c8004e10 con 0x7f04d01470b0 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.924+0000 7f04b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.924+0000 7f04b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.924+0000 7f04b7fff640 10 monclient: dump: 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:09.926 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:09.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.924+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f04c801ae60 con 0x7f04d01470b0 2024-03-29T11:38:09.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.924+0000 7f04cd7fa640 4 set_mon_vals no callback set 2024-03-29T11:38:09.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.925+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f04c8084d70 con 0x7f04d01470b0 2024-03-29T11:38:09.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.925+0000 7f04b7fff640 10 client.4305.objecter ms_dispatch 0x7f04d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:09.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.925+0000 7f04b7fff640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:09.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.925+0000 7f04b7fff640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:09.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.925+0000 7f04b7fff640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-29T11:38:09.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.925+0000 7f04cf7fe640 10 monclient: get_auth_request con 0x7f04bc06dd70 auth_method 0 2024-03-29T11:38:09.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.926+0000 7f04b7fff640 10 client.4305.objecter ms_handle_connect 0x7f04bc06dd70 2024-03-29T11:38:09.933 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.933+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04c8050500 con 0x7f04d01470b0 2024-03-29T11:38:09.933 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.933+0000 7f04b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:09.933 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:09.933+0000 7f04b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:10.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.267+0000 7f04b5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:10.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.267+0000 7f04b5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:10.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.267+0000 7f04b5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:38:10.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.267+0000 7f04b5ffb640 1 -- 172.21.15.100:0/726965765 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04d0075f10 con 0x7f04d01470b0 2024-03-29T11:38:10.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.267+0000 7f04b7fff640 1 -- 172.21.15.100:0/726965765 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f04c8052600 con 0x7f04d01470b0 2024-03-29T11:38:10.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.267+0000 7f04b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:10.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.267+0000 7f04b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:10.268 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:38:10.268 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:10.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.271+0000 7f04b5ffb640 10 librados: watch_flush enter 2024-03-29T11:38:10.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.271+0000 7f04b5ffb640 10 librados: watch_flush exit 2024-03-29T11:38:10.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.272+0000 7f04b5ffb640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-29T11:38:10.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.272+0000 7f04b5ffb640 10 client.4305.objecter successfully canceled tick 2024-03-29T11:38:10.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.272+0000 7f04b5ffb640 1 -- 172.21.15.100:0/726965765 >> v1:172.21.15.100:6816/3220289657 conn(0x7f04bc06dd70 legacy=0x7f04bc070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:10.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.272+0000 7f04b5ffb640 10 monclient: shutdown 2024-03-29T11:38:10.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.272+0000 7f04b5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:10.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.272+0000 7f04b5ffb640 1 -- 172.21.15.100:0/726965765 >> v1:172.21.15.100:6789/0 conn(0x7f04d01470b0 legacy=0x7f04d01203a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:10.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.273+0000 7f04b5ffb640 1 -- 172.21.15.100:0/726965765 shutdown_connections 2024-03-29T11:38:10.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.273+0000 7f04b5ffb640 1 -- 172.21.15.100:0/726965765 >> 172.21.15.100:0/726965765 conn(0x7f04d0066a00 msgr2=0x7f04d0142f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:10.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.273+0000 7f04b5ffb640 1 -- 172.21.15.100:0/726965765 shutdown_connections 2024-03-29T11:38:10.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.273+0000 7f04b5ffb640 1 -- 172.21.15.100:0/726965765 wait complete. 2024-03-29T11:38:10.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:10.273+0000 7f04b5ffb640 1 librados: shutdown 2024-03-29T11:38:10.290 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:17.098 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:17.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.245+0000 7fe60a5a3640 10 monclient: get_monmap_and_config 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.245+0000 7fe60a5a3640 10 monclient: build_initial_monmap 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.245+0000 7fe60a5a3640 10 monclient: monmap: 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:38:17.246378+0000 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:38:17.246378+0000 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:38:17.247 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:17.248 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.247+0000 7fe60a5a3640 1 Processor -- start 2024-03-29T11:38:17.248 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.247+0000 7fe60a5a3640 1 -- start start 2024-03-29T11:38:17.248 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.247+0000 7fe60a5a3640 10 monclient: init 2024-03-29T11:38:17.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.248+0000 7fe60a5a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:17.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.248+0000 7fe60a5a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:17.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.248+0000 7fe60a5a3640 10 monclient(hunting): picked mon.noname-a con 0x7fe6040a9ac0 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:17.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.248+0000 7fe60a5a3640 10 monclient(hunting): picked mon.noname-b con 0x7fe6040b4100 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:17.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.248+0000 7fe60a5a3640 10 monclient(hunting): picked mon.noname-c con 0x7fe60413ee90 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:17.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.248+0000 7fe60a5a3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6040af6f0 con 0x7fe6040a9ac0 2024-03-29T11:38:17.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.249+0000 7fe60a5a3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6040aac80 con 0x7fe60413ee90 2024-03-29T11:38:17.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.249+0000 7fe60a5a3640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6040aaf60 con 0x7fe6040b4100 2024-03-29T11:38:17.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.249+0000 7fe60a5a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:17.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.249+0000 7fe60a5a3640 10 monclient(hunting): authenticate will time out at 1017.444266s 2024-03-29T11:38:17.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.252+0000 7fe608b40640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe60413ee90 0x7fe60414a210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:47109/0 (socket says 172.21.15.100:47109) 2024-03-29T11:38:17.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.252+0000 7fe608b40640 1 -- 172.21.15.100:0/3263174021 learned_addr learned my addr 172.21.15.100:0/3263174021 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:17.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.255+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973702985 0 0) 0x7fe6040aac80 con 0x7fe60413ee90 2024-03-29T11:38:17.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:17.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:17.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:17.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:38:17.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ec003520 con 0x7fe60413ee90 2024-03-29T11:38:17.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659055830 0 0) 0x7fe6040aaf60 con 0x7fe6040b4100 2024-03-29T11:38:17.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.256+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6040aac80 con 0x7fe6040b4100 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2797042242 0 0) 0x7fe6040aac80 con 0x7fe6040b4100 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 >> v1:172.21.15.191:6789/0 conn(0x7fe60413ee90 legacy=0x7fe60414a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 >> v1:172.21.15.100:6789/0 conn(0x7fe6040a9ac0 legacy=0x7fe6040adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 1 monclient: found mon.noname-b 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:17.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60413e970 con 0x7fe6040b4100 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.257+0000 7fe602ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe5fc001970 con 0x7fe6040b4100 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe602ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe602ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe602ffd640 10 monclient: dump: 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:17.262 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5fc001bc0 con 0x7fe6040b4100 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe60a5a3640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe60a5a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe602ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe60a5a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe60a5a3640 4 set_mon_vals no callback set 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe602ffd640 1 -- 172.21.15.100:0/3263174021 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe5fc001f50 con 0x7fe6040b4100 2024-03-29T11:38:17.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe602ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe602ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.258+0000 7fe602ffd640 10 monclient: dump: 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.259+0000 7fe60a5a3640 10 monclient: shutdown 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.259+0000 7fe60a5a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:17.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.259+0000 7fe60a5a3640 1 -- 172.21.15.100:0/3263174021 >> v1:172.21.15.100:6790/0 conn(0x7fe6040b4100 legacy=0x7fe604146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:17.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.259+0000 7fe60a5a3640 1 -- 172.21.15.100:0/3263174021 shutdown_connections 2024-03-29T11:38:17.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.259+0000 7fe60a5a3640 1 -- 172.21.15.100:0/3263174021 >> 172.21.15.100:0/3263174021 conn(0x7fe6040b35e0 msgr2=0x7fe6040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:17.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.259+0000 7fe60a5a3640 1 -- 172.21.15.100:0/3263174021 shutdown_connections 2024-03-29T11:38:17.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.259+0000 7fe60a5a3640 1 -- 172.21.15.100:0/3263174021 wait complete. 2024-03-29T11:38:17.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.260+0000 7fe60a5a3640 10 monclient: build_initial_monmap 2024-03-29T11:38:17.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.260+0000 7fe60a5a3640 10 monclient: monmap: 2024-03-29T11:38:17.265 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:17.265 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:17.265 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:38:17.265 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.260+0000 7fe60a5a3640 1 librados: starting msgr at 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.260+0000 7fe60a5a3640 1 librados: starting objecter 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.260+0000 7fe60a5a3640 1 Processor -- start 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.261+0000 7fe60a5a3640 1 -- start start 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.261+0000 7fe60a5a3640 1 librados: setting wanted keys 2024-03-29T11:38:17.266 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.261+0000 7fe60a5a3640 1 librados: calling monclient init 2024-03-29T11:38:17.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.261+0000 7fe60a5a3640 10 monclient: init 2024-03-29T11:38:17.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.261+0000 7fe60a5a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:17.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.261+0000 7fe60a5a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:17.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.261+0000 7fe60a5a3640 10 monclient(hunting): picked mon.noname-a con 0x7fe6040a9ac0 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:17.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.264+0000 7fe60a5a3640 10 monclient(hunting): picked mon.noname-b con 0x7fe6040b4100 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:17.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.265+0000 7fe60a5a3640 10 monclient(hunting): picked mon.noname-c con 0x7fe60413ee90 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:17.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.265+0000 7fe60a5a3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe604135be0 con 0x7fe6040a9ac0 2024-03-29T11:38:17.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.265+0000 7fe60a5a3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6041c51d0 con 0x7fe6040b4100 2024-03-29T11:38:17.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.265+0000 7fe60a5a3640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6041c54b0 con 0x7fe60413ee90 2024-03-29T11:38:17.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.265+0000 7fe60a5a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:17.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.265+0000 7fe60a5a3640 10 monclient(hunting): authenticate will time out at 1017.460192s 2024-03-29T11:38:17.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.268+0000 7fe6037fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe6040b4100 0x7fe604134810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:47189/0 (socket says 172.21.15.100:47189) 2024-03-29T11:38:17.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.268+0000 7fe6037fe640 1 -- 172.21.15.100:0/76573891 learned_addr learned my addr 172.21.15.100:0/76573891 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:17.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.272+0000 7fe600ff9640 10 client.?.objecter ms_handle_connect 0x7fe60413ee90 2024-03-29T11:38:17.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.273+0000 7fe600ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:17.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.273+0000 7fe600ff9640 10 client.?.objecter ms_handle_connect 0x7fe6040b4100 2024-03-29T11:38:17.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.273+0000 7fe600ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:17.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.273+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121458271 0 0) 0x7fe6041c54b0 con 0x7fe60413ee90 2024-03-29T11:38:17.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.273+0000 7fe600ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:17.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.273+0000 7fe600ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:17.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.273+0000 7fe600ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:17.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.273+0000 7fe600ff9640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:38:17.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.273+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f0004eb0 con 0x7fe60413ee90 2024-03-29T11:38:17.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.274+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748193444 0 0) 0x7fe6041c51d0 con 0x7fe6040b4100 2024-03-29T11:38:17.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.274+0000 7fe600ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:17.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.274+0000 7fe600ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:17.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.274+0000 7fe600ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.274+0000 7fe600ff9640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.274+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6041c54b0 con 0x7fe6040b4100 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.275+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 971292393 0 0) 0x7fe5f0004eb0 con 0x7fe60413ee90 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.275+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6041c51d0 con 0x7fe60413ee90 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.275+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe5f8001810 con 0x7fe60413ee90 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.275+0000 7fe600ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.275+0000 7fe600ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.275+0000 7fe600ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:17.276 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:17.277 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:17.277 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:17.277 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:17.277 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:17.277 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.275+0000 7fe600ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:17.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4105485768 0 0) 0x7fe6041c54b0 con 0x7fe6040b4100 2024-03-29T11:38:17.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5f0004eb0 con 0x7fe6040b4100 2024-03-29T11:38:17.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe5fc001ed0 con 0x7fe6040b4100 2024-03-29T11:38:17.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:17.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:17.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:17.280 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:17.280 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4101986754 0 0) 0x7fe5f0004eb0 con 0x7fe6040b4100 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 >> v1:172.21.15.100:6790/0 conn(0x7fe60413ee90 legacy=0x7fe60406edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 >> v1:172.21.15.100:6789/0 conn(0x7fe6040a9ac0 legacy=0x7fe604072090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 1 monclient: found mon.b 2024-03-29T11:38:17.281 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe604071280 con 0x7fe6040b4100 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.276+0000 7fe600ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.277+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5fc003c20 con 0x7fe6040b4100 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.277+0000 7fe600ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.277+0000 7fe6027fc640 4 set_mon_vals no callback set 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe5fc004030 con 0x7fe6040b4100 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe600ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe600ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe600ff9640 10 monclient: dump: 2024-03-29T11:38:17.282 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe60a5a3640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe60a5a3640 10 monclient: _renew_subs 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe60a5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe60a5a3640 1 -- 172.21.15.100:0/76573891 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe604134f20 con 0x7fe6040b4100 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe60a5a3640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:17.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe60a5a3640 10 monclient: _renew_subs 2024-03-29T11:38:17.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe60a5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:38:17.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe60a5a3640 1 -- 172.21.15.100:0/76573891 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60406f4c0 con 0x7fe6040b4100 2024-03-29T11:38:17.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.278+0000 7fe60a5a3640 1 librados: init done 2024-03-29T11:38:17.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.280+0000 7fe60a5a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:17.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.281+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fe5fc004190 con 0x7fe6040b4100 2024-03-29T11:38:17.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.281+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fe5fc084f00 con 0x7fe6040b4100 2024-03-29T11:38:17.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.281+0000 7fe600ff9640 10 client.4306.objecter ms_dispatch 0x7fe604000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:17.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.281+0000 7fe600ff9640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:17.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.281+0000 7fe600ff9640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:17.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.282+0000 7fe600ff9640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T11:38:17.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.282+0000 7fe60a5a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:17.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.282+0000 7fe60a5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:38:17.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.282+0000 7fe60a5a3640 1 -- 172.21.15.100:0/76573891 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60406ead0 con 0x7fe6040b4100 2024-03-29T11:38:17.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.282+0000 7fe603fff640 10 monclient: get_auth_request con 0x7fe5f006de30 auth_method 0 2024-03-29T11:38:17.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.282+0000 7fe600ff9640 10 client.4306.objecter ms_handle_connect 0x7fe5f006de30 2024-03-29T11:38:17.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.290+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe60406ead0 con 0x7fe6040b4100 2024-03-29T11:38:17.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.290+0000 7fe600ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:17.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.290+0000 7fe600ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:17.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.604+0000 7fe60a5a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:17.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.604+0000 7fe60a5a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:17.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.604+0000 7fe60a5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:38:17.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.604+0000 7fe60a5a3640 1 -- 172.21.15.100:0/76573891 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe60406ead0 con 0x7fe6040b4100 2024-03-29T11:38:17.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.608+0000 7fe600ff9640 1 -- 172.21.15.100:0/76573891 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fe60406ead0 con 0x7fe6040b4100 2024-03-29T11:38:17.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.608+0000 7fe600ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:17.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.608+0000 7fe600ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:17.609 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:38:17.609 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:17.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.612+0000 7fe60a5a3640 10 librados: watch_flush enter 2024-03-29T11:38:17.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.612+0000 7fe60a5a3640 10 librados: watch_flush exit 2024-03-29T11:38:17.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.612+0000 7fe60a5a3640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T11:38:17.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.612+0000 7fe60a5a3640 10 client.4306.objecter successfully canceled tick 2024-03-29T11:38:17.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.612+0000 7fe60a5a3640 1 -- 172.21.15.100:0/76573891 >> v1:172.21.15.100:6816/3220289657 conn(0x7fe5f006de30 legacy=0x7fe5f00702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:17.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.612+0000 7fe60a5a3640 10 monclient: shutdown 2024-03-29T11:38:17.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.612+0000 7fe60a5a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:17.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.612+0000 7fe60a5a3640 1 -- 172.21.15.100:0/76573891 >> v1:172.21.15.191:6789/0 conn(0x7fe6040b4100 legacy=0x7fe604134810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:17.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.614+0000 7fe60a5a3640 1 -- 172.21.15.100:0/76573891 shutdown_connections 2024-03-29T11:38:17.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.614+0000 7fe60a5a3640 1 -- 172.21.15.100:0/76573891 >> 172.21.15.100:0/76573891 conn(0x7fe6040b35e0 msgr2=0x7fe6040b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:17.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.615+0000 7fe60a5a3640 1 -- 172.21.15.100:0/76573891 shutdown_connections 2024-03-29T11:38:17.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.615+0000 7fe60a5a3640 1 -- 172.21.15.100:0/76573891 wait complete. 2024-03-29T11:38:17.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:17.615+0000 7fe60a5a3640 1 librados: shutdown 2024-03-29T11:38:17.632 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:24.439 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:24.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.593+0000 7f27b87f6640 10 monclient: get_monmap_and_config 2024-03-29T11:38:24.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.593+0000 7f27b87f6640 10 monclient: build_initial_monmap 2024-03-29T11:38:24.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.593+0000 7f27b87f6640 10 monclient: monmap: 2024-03-29T11:38:24.595 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:24.595 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:24.596 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:38:24.594669+0000 2024-03-29T11:38:24.596 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:38:24.594669+0000 2024-03-29T11:38:24.596 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:24.596 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:24.596 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:24.596 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:38:24.596 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:38:24.596 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:24.598 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.596+0000 7f27b87f6640 1 Processor -- start 2024-03-29T11:38:24.598 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.596+0000 7f27b87f6640 1 -- start start 2024-03-29T11:38:24.598 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.596+0000 7f27b87f6640 10 monclient: init 2024-03-29T11:38:24.598 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.597+0000 7f27b87f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:24.598 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.597+0000 7f27b87f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:24.598 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.597+0000 7f27b87f6640 10 monclient(hunting): picked mon.noname-c con 0x7f27b00a9930 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:24.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.597+0000 7f27b87f6640 10 monclient(hunting): picked mon.noname-a con 0x7f27b0149dc0 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:24.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.597+0000 7f27b87f6640 10 monclient(hunting): picked mon.noname-b con 0x7f27b014ff40 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:24.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.597+0000 7f27b87f6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b00aae60 con 0x7f27b0149dc0 2024-03-29T11:38:24.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.597+0000 7f27b87f6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b00b4310 con 0x7f27b00a9930 2024-03-29T11:38:24.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.597+0000 7f27b87f6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b00afdd0 con 0x7f27b014ff40 2024-03-29T11:38:24.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.597+0000 7f27b87f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:24.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.597+0000 7f27b87f6640 10 monclient(hunting): authenticate will time out at 1024.792343s 2024-03-29T11:38:24.603 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.602+0000 7f27b6592640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f27b00a9930 0x7f27b0143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:52849/0 (socket says 172.21.15.100:52849) 2024-03-29T11:38:24.603 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.602+0000 7f27b6592640 1 -- 172.21.15.100:0/2036911841 learned_addr learned my addr 172.21.15.100:0/2036911841 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:24.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.603+0000 7f27b5590640 1 -- 172.21.15.100:0/2036911841 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082569376 0 0) 0x7f27b00b4310 con 0x7f27b00a9930 2024-03-29T11:38:24.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.603+0000 7f27b5590640 10 monclient(hunting): _negotiate 2024-03-29T11:38:24.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.603+0000 7f27b5590640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:24.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.603+0000 7f27b5590640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:24.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.603+0000 7f27b5590640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:38:24.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.603+0000 7f27b5590640 1 -- 172.21.15.100:0/2036911841 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f279c003520 con 0x7f27b00a9930 2024-03-29T11:38:24.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.603+0000 7f27b5590640 1 -- 172.21.15.100:0/2036911841 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1784825380 0 0) 0x7f279c003520 con 0x7f27b00a9930 2024-03-29T11:38:24.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.603+0000 7f27b5590640 1 -- 172.21.15.100:0/2036911841 >> v1:172.21.15.100:6790/0 conn(0x7f27b014ff40 legacy=0x7f27b0152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:24.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 1 -- 172.21.15.100:0/2036911841 >> v1:172.21.15.100:6789/0 conn(0x7f27b0149dc0 legacy=0x7f27b014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:24.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 10 monclient: _finish_hunting 0 2024-03-29T11:38:24.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 1 monclient: found mon.noname-c 2024-03-29T11:38:24.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:24.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:38:24.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 1 -- 172.21.15.100:0/2036911841 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b014a950 con 0x7f27b00a9930 2024-03-29T11:38:24.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 10 monclient: _finish_auth 0 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 10 monclient: _check_auth_tickets 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 1 -- 172.21.15.100:0/2036911841 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f27a0001690 con 0x7f27b00a9930 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.605+0000 7f27b5590640 10 monclient: dump: 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:24.608 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:24.609 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:24.609 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:24.609 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:24.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.606+0000 7f27b87f6640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:38:24.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.606+0000 7f27b87f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:24.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.606+0000 7f27b5590640 1 -- 172.21.15.100:0/2036911841 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a00018f0 con 0x7f27b00a9930 2024-03-29T11:38:24.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.606+0000 7f27b5590640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:24.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.606+0000 7f27b87f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:24.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.606+0000 7f27b87f6640 4 set_mon_vals no callback set 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.607+0000 7f27b5590640 1 -- 172.21.15.100:0/2036911841 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f27a0001d40 con 0x7f27b00a9930 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.607+0000 7f27b5590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.607+0000 7f27b5590640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.607+0000 7f27b5590640 10 monclient: dump: 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:24.611 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:24.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.607+0000 7f27b87f6640 10 monclient: shutdown 2024-03-29T11:38:24.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.607+0000 7f27b87f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:24.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.607+0000 7f27b87f6640 1 -- 172.21.15.100:0/2036911841 >> v1:172.21.15.191:6789/0 conn(0x7f27b00a9930 legacy=0x7f27b0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:24.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.609+0000 7f27b87f6640 1 -- 172.21.15.100:0/2036911841 shutdown_connections 2024-03-29T11:38:24.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.609+0000 7f27b87f6640 1 -- 172.21.15.100:0/2036911841 >> 172.21.15.100:0/2036911841 conn(0x7f27b00b35e0 msgr2=0x7f27b00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:24.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.609+0000 7f27b87f6640 1 -- 172.21.15.100:0/2036911841 shutdown_connections 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.609+0000 7f27b87f6640 1 -- 172.21.15.100:0/2036911841 wait complete. 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.610+0000 7f27b87f6640 10 monclient: build_initial_monmap 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.610+0000 7f27b87f6640 10 monclient: monmap: 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:24.613 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.611+0000 7f27b87f6640 1 librados: starting msgr at 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.611+0000 7f27b87f6640 1 librados: starting objecter 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.611+0000 7f27b87f6640 1 Processor -- start 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.611+0000 7f27b87f6640 1 -- start start 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.611+0000 7f27b87f6640 1 librados: setting wanted keys 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.611+0000 7f27b87f6640 1 librados: calling monclient init 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.611+0000 7f27b87f6640 10 monclient: init 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.612+0000 7f27b87f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:24.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.612+0000 7f27b87f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:24.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.612+0000 7f27b87f6640 10 monclient(hunting): picked mon.noname-c con 0x7f27b00a9930 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:24.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.612+0000 7f27b87f6640 10 monclient(hunting): picked mon.noname-b con 0x7f27b0149dc0 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:24.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.612+0000 7f27b87f6640 10 monclient(hunting): picked mon.noname-a con 0x7f27b014ff40 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:24.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.612+0000 7f27b87f6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b01dc360 con 0x7f27b014ff40 2024-03-29T11:38:24.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.612+0000 7f27b87f6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b013e930 con 0x7f27b0149dc0 2024-03-29T11:38:24.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.612+0000 7f27b87f6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b013e570 con 0x7f27b00a9930 2024-03-29T11:38:24.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.612+0000 7f27b87f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:24.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.612+0000 7f27b87f6640 10 monclient(hunting): authenticate will time out at 1024.807376s 2024-03-29T11:38:24.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.615+0000 7f27b6592640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f27b00a9930 0x7f27b01dbc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:59981/0 (socket says 172.21.15.100:59981) 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.615+0000 7f27b6592640 1 -- 172.21.15.100:0/2937867929 learned_addr learned my addr 172.21.15.100:0/2937867929 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 10 client.?.objecter ms_handle_connect 0x7f27b00a9930 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326579803 0 0) 0x7f27b013e570 con 0x7f27b00a9930 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2794004eb0 con 0x7f27b00a9930 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2889838951 0 0) 0x7f2794004eb0 con 0x7f27b00a9930 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27b013e570 con 0x7f27b00a9930 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f27a00014b0 con 0x7f27b00a9930 2024-03-29T11:38:24.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 671511737 0 0) 0x7f27b013e570 con 0x7f27b00a9930 2024-03-29T11:38:24.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 >> v1:172.21.15.191:6789/0 conn(0x7f27b0149dc0 legacy=0x7f27b01383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:24.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.616+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 >> v1:172.21.15.100:6789/0 conn(0x7f27b014ff40 legacy=0x7f27b013ee50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27a6ffd640 1 monclient: found mon.c 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27a6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b013e700 con 0x7f27b00a9930 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27a6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27b87f6640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27b87f6640 10 monclient: _renew_subs 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27b87f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:24.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27b87f6640 1 -- 172.21.15.100:0/2937867929 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b0138ad0 con 0x7f27b00a9930 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.619+0000 7f27b87f6640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27b87f6640 10 monclient: _renew_subs 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27b87f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27b87f6640 1 -- 172.21.15.100:0/2937867929 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b01db860 con 0x7f27b00a9930 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27b87f6640 1 librados: init done 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a0004c70 con 0x7f27b00a9930 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f27a0001610 con 0x7f27b00a9930 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27a6ffd640 10 monclient: dump: 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:24.624 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.620+0000 7f27b4d8f640 4 set_mon_vals no callback set 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.622+0000 7f27b87f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.622+0000 7f27b87f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.622+0000 7f27b87f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:24.625 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.622+0000 7f27b87f6640 1 -- 172.21.15.100:0/2937867929 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2798005080 con 0x7f27b00a9930 2024-03-29T11:38:24.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.623+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f27a0004f80 con 0x7f27b00a9930 2024-03-29T11:38:24.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.624+0000 7f27b5d91640 10 monclient: get_auth_request con 0x7f279406dd20 auth_method 0 2024-03-29T11:38:24.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.624+0000 7f27a6ffd640 10 client.4298.objecter ms_handle_connect 0x7f279406dd20 2024-03-29T11:38:24.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.624+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f27a0084d60 con 0x7f27b00a9930 2024-03-29T11:38:24.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.624+0000 7f27a6ffd640 10 client.4298.objecter ms_dispatch 0x7f27b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:24.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.624+0000 7f27a6ffd640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:24.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.624+0000 7f27a6ffd640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:24.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.624+0000 7f27a6ffd640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T11:38:24.632 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.632+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27a0050640 con 0x7f27b00a9930 2024-03-29T11:38:24.632 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.632+0000 7f27a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:24.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.632+0000 7f27a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:24.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.953+0000 7f27b87f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:24.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.953+0000 7f27b87f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:24.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.953+0000 7f27b87f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:24.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.953+0000 7f27b87f6640 1 -- 172.21.15.100:0/2937867929 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2798005330 con 0x7f27b00a9930 2024-03-29T11:38:24.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.958+0000 7f27a6ffd640 1 -- 172.21.15.100:0/2937867929 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f27a0052570 con 0x7f27b00a9930 2024-03-29T11:38:24.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.958+0000 7f27a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:24.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.958+0000 7f27a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:24.963 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:38:24.964 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 10 librados: watch_flush enter 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 10 librados: watch_flush exit 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 10 client.4298.objecter successfully canceled tick 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 1 -- 172.21.15.100:0/2937867929 >> v1:172.21.15.100:6816/3220289657 conn(0x7f279406dd20 legacy=0x7f27940701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 10 monclient: shutdown 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 1 -- 172.21.15.100:0/2937867929 >> v1:172.21.15.100:6790/0 conn(0x7f27b00a9930 legacy=0x7f27b01dbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 1 -- 172.21.15.100:0/2937867929 shutdown_connections 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 1 -- 172.21.15.100:0/2937867929 >> 172.21.15.100:0/2937867929 conn(0x7f27b00b35e0 msgr2=0x7f27b00b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 1 -- 172.21.15.100:0/2937867929 shutdown_connections 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 1 -- 172.21.15.100:0/2937867929 wait complete. 2024-03-29T11:38:24.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:24.966+0000 7f27b87f6640 1 librados: shutdown 2024-03-29T11:38:24.985 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:31.792 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:31.941 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.941+0000 7f5ebce3e640 10 monclient: get_monmap_and_config 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.941+0000 7f5ebce3e640 10 monclient: build_initial_monmap 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.941+0000 7f5ebce3e640 10 monclient: monmap: 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:38:31.942234+0000 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:38:31.942234+0000 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:38:31.942 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:38:31.943 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:31.946 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.944+0000 7f5ebce3e640 1 Processor -- start 2024-03-29T11:38:31.946 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.944+0000 7f5ebce3e640 1 -- start start 2024-03-29T11:38:31.946 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.944+0000 7f5ebce3e640 10 monclient: init 2024-03-29T11:38:31.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.946+0000 7f5ebce3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:31.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.946+0000 7f5ebce3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:31.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.946+0000 7f5ebce3e640 10 monclient(hunting): picked mon.noname-a con 0x7f5eb80b1f60 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:31.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.946+0000 7f5ebce3e640 10 monclient(hunting): picked mon.noname-c con 0x7f5eb8071eb0 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:31.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.946+0000 7f5ebce3e640 10 monclient(hunting): picked mon.noname-b con 0x7f5eb8143d10 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:31.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.946+0000 7f5ebce3e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb806ef80 con 0x7f5eb80b1f60 2024-03-29T11:38:31.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.946+0000 7f5ebce3e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb806f260 con 0x7f5eb8071eb0 2024-03-29T11:38:31.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.946+0000 7f5ebce3e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80b40c0 con 0x7f5eb8143d10 2024-03-29T11:38:31.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.946+0000 7f5ebce3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:31.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.946+0000 7f5ebce3e640 10 monclient(hunting): authenticate will time out at 1032.141283s 2024-03-29T11:38:31.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.954+0000 7f5eb659c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f5eb80b1f60 0x7f5eb8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:50421/0 (socket says 172.21.15.100:50421) 2024-03-29T11:38:31.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.954+0000 7f5eb659c640 1 -- 172.21.15.100:0/2533720248 learned_addr learned my addr 172.21.15.100:0/2533720248 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:31.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.955+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490132361 0 0) 0x7f5eb806ef80 con 0x7f5eb80b1f60 2024-03-29T11:38:31.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:31.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:31.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:31.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:38:31.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea0003520 con 0x7f5eb80b1f60 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199934186 0 0) 0x7f5eb806f260 con 0x7f5eb8071eb0 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb806ef80 con 0x7f5eb8071eb0 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980352602 0 0) 0x7f5eb80b40c0 con 0x7f5eb8143d10 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:38:31.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb806f260 con 0x7f5eb8143d10 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1236834087 0 0) 0x7f5eb806f260 con 0x7f5eb8143d10 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 >> v1:172.21.15.191:6789/0 conn(0x7f5eb8071eb0 legacy=0x7f5eb8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 >> v1:172.21.15.100:6789/0 conn(0x7f5eb80b1f60 legacy=0x7f5eb8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 monclient: found mon.noname-b 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8155d70 con 0x7f5eb8143d10 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient: _finish_auth 0 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5eb00017e0 con 0x7f5eb8143d10 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:31.959 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5eb559a640 10 monclient: dump: 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5ebce3e640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:38:31.960 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.956+0000 7f5ebce3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eb00019c0 con 0x7f5eb8143d10 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5eb559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5eb559a640 1 -- 172.21.15.100:0/2533720248 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5eb0001e10 con 0x7f5eb8143d10 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5eb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5eb559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5eb559a640 10 monclient: dump: 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:31.961 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:31.962 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:31.962 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:31.962 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:31.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5ebce3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:31.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5ebce3e640 4 set_mon_vals no callback set 2024-03-29T11:38:31.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5ebce3e640 10 monclient: shutdown 2024-03-29T11:38:31.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5ebce3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:31.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.957+0000 7f5ebce3e640 1 -- 172.21.15.100:0/2533720248 >> v1:172.21.15.100:6790/0 conn(0x7f5eb8143d10 legacy=0x7f5eb8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:31.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.960+0000 7f5ebce3e640 1 -- 172.21.15.100:0/2533720248 shutdown_connections 2024-03-29T11:38:31.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.960+0000 7f5ebce3e640 1 -- 172.21.15.100:0/2533720248 >> 172.21.15.100:0/2533720248 conn(0x7f5eb8066a00 msgr2=0x7f5eb8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:31.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.961+0000 7f5ebce3e640 1 -- 172.21.15.100:0/2533720248 shutdown_connections 2024-03-29T11:38:31.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.961+0000 7f5ebce3e640 1 -- 172.21.15.100:0/2533720248 wait complete. 2024-03-29T11:38:31.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.962+0000 7f5ebce3e640 10 monclient: build_initial_monmap 2024-03-29T11:38:31.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.962+0000 7f5ebce3e640 10 monclient: monmap: 2024-03-29T11:38:31.965 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:31.965 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:31.965 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:38:31.965 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:38:31.965 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:31.965 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:31.965 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.962+0000 7f5ebce3e640 1 librados: starting msgr at 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.962+0000 7f5ebce3e640 1 librados: starting objecter 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.962+0000 7f5ebce3e640 1 Processor -- start 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.962+0000 7f5ebce3e640 1 -- start start 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.962+0000 7f5ebce3e640 1 librados: setting wanted keys 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.962+0000 7f5ebce3e640 1 librados: calling monclient init 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.962+0000 7f5ebce3e640 10 monclient: init 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.963+0000 7f5ebce3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.963+0000 7f5ebce3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.963+0000 7f5ebce3e640 10 monclient(hunting): picked mon.noname-a con 0x7f5eb80b1f60 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:31.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.963+0000 7f5ebce3e640 10 monclient(hunting): picked mon.noname-b con 0x7f5eb8143d10 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:31.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.963+0000 7f5ebce3e640 10 monclient(hunting): picked mon.noname-c con 0x7f5eb81464d0 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:31.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.963+0000 7f5ebce3e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80b14b0 con 0x7f5eb80b1f60 2024-03-29T11:38:31.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.963+0000 7f5ebce3e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80ac590 con 0x7f5eb8143d10 2024-03-29T11:38:31.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.963+0000 7f5ebce3e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80ab4a0 con 0x7f5eb81464d0 2024-03-29T11:38:31.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.963+0000 7f5ebce3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:31.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.963+0000 7f5ebce3e640 10 monclient(hunting): authenticate will time out at 1032.158815s 2024-03-29T11:38:31.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5eb6d9d640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5eb81464d0 0x7f5eb81ee8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:49883/0 (socket says 172.21.15.100:49883) 2024-03-29T11:38:31.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5eb6d9d640 1 -- 172.21.15.100:0/1639915121 learned_addr learned my addr 172.21.15.100:0/1639915121 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:31.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 10 client.?.objecter ms_handle_connect 0x7f5eb81464d0 2024-03-29T11:38:31.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:31.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507328608 0 0) 0x7f5eb80ab4a0 con 0x7f5eb81464d0 2024-03-29T11:38:31.974 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:31.974 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:31.974 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:31.974 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:38:31.974 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e98004eb0 con 0x7f5eb81464d0 2024-03-29T11:38:31.974 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2866203192 0 0) 0x7f5e98004eb0 con 0x7f5eb81464d0 2024-03-29T11:38:31.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eb80ab4a0 con 0x7f5eb81464d0 2024-03-29T11:38:31.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5eb0001770 con 0x7f5eb81464d0 2024-03-29T11:38:31.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:31.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.970+0000 7f5ea6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:31.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3946219433 0 0) 0x7f5eb80ab4a0 con 0x7f5eb81464d0 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 >> v1:172.21.15.191:6789/0 conn(0x7f5eb8143d10 legacy=0x7f5eb8145dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 >> v1:172.21.15.100:6789/0 conn(0x7f5eb80b1f60 legacy=0x7f5eb80aad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 1 monclient: found mon.c 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8146940 con 0x7f5eb81464d0 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ebce3e640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ebce3e640 10 monclient: _renew_subs 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ebce3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:31.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ebce3e640 1 -- 172.21.15.100:0/1639915121 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb80aa1a0 con 0x7f5eb81464d0 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ebce3e640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ebce3e640 10 monclient: _renew_subs 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ebce3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ebce3e640 1 -- 172.21.15.100:0/1639915121 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb81f01c0 con 0x7f5eb81464d0 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ebce3e640 1 librados: init done 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5eb0003c20 con 0x7f5eb81464d0 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5eb00013a0 con 0x7f5eb81464d0 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.971+0000 7f5ea6ffd640 10 monclient: dump: 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:31.980 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.974+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f5eb0004070 con 0x7f5eb81464d0 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.975+0000 7f5ea7fff640 4 set_mon_vals no callback set 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.975+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f5eb0084ee0 con 0x7f5eb81464d0 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.975+0000 7f5ea6ffd640 10 client.4304.objecter ms_dispatch 0x7f5eb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.975+0000 7f5ea6ffd640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.975+0000 7f5ea6ffd640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.975+0000 7f5ea6ffd640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T11:38:31.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.976+0000 7f5eb659c640 10 monclient: get_auth_request con 0x7f5e9806dc50 auth_method 0 2024-03-29T11:38:31.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.976+0000 7f5ea6ffd640 10 client.4304.objecter ms_handle_connect 0x7f5e9806dc50 2024-03-29T11:38:31.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.976+0000 7f5ebce3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:31.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.976+0000 7f5ebce3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:31.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.976+0000 7f5ebce3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:31.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.976+0000 7f5ebce3e640 1 -- 172.21.15.100:0/1639915121 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c005080 con 0x7f5eb81464d0 2024-03-29T11:38:31.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.989+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5eb00507c0 con 0x7f5eb81464d0 2024-03-29T11:38:31.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.989+0000 7f5ea6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:31.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:31.989+0000 7f5ea6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:32.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.312+0000 7f5ebce3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:32.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.312+0000 7f5ebce3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:32.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.312+0000 7f5ebce3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:32.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.312+0000 7f5ebce3e640 1 -- 172.21.15.100:0/1639915121 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e9c005330 con 0x7f5eb81464d0 2024-03-29T11:38:32.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.315+0000 7f5ea6ffd640 1 -- 172.21.15.100:0/1639915121 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f5eb00526f0 con 0x7f5eb81464d0 2024-03-29T11:38:32.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.315+0000 7f5ea6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:32.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.315+0000 7f5ea6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:32.316 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:38:32.316 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:32.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.318+0000 7f5ebce3e640 10 librados: watch_flush enter 2024-03-29T11:38:32.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.318+0000 7f5ebce3e640 10 librados: watch_flush exit 2024-03-29T11:38:32.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.318+0000 7f5ebce3e640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T11:38:32.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.318+0000 7f5ebce3e640 10 client.4304.objecter successfully canceled tick 2024-03-29T11:38:32.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.318+0000 7f5ebce3e640 1 -- 172.21.15.100:0/1639915121 >> v1:172.21.15.100:6816/3220289657 conn(0x7f5e9806dc50 legacy=0x7f5e98070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:32.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.318+0000 7f5ebce3e640 10 monclient: shutdown 2024-03-29T11:38:32.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.318+0000 7f5ebce3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:32.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.318+0000 7f5ebce3e640 1 -- 172.21.15.100:0/1639915121 >> v1:172.21.15.100:6790/0 conn(0x7f5eb81464d0 legacy=0x7f5eb81ee8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:32.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.319+0000 7f5ebce3e640 1 -- 172.21.15.100:0/1639915121 shutdown_connections 2024-03-29T11:38:32.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.319+0000 7f5ebce3e640 1 -- 172.21.15.100:0/1639915121 >> 172.21.15.100:0/1639915121 conn(0x7f5eb8066a00 msgr2=0x7f5eb80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:32.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.322+0000 7f5ebce3e640 1 -- 172.21.15.100:0/1639915121 shutdown_connections 2024-03-29T11:38:32.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.322+0000 7f5ebce3e640 1 -- 172.21.15.100:0/1639915121 wait complete. 2024-03-29T11:38:32.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:32.322+0000 7f5ebce3e640 1 librados: shutdown 2024-03-29T11:38:32.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:39.147 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:39.278 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.277+0000 7fe41c36b640 10 monclient: get_monmap_and_config 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.277+0000 7fe41c36b640 10 monclient: build_initial_monmap 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.278+0000 7fe41c36b640 10 monclient: monmap: 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:38:39.279008+0000 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:38:39.279008+0000 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:38:39.279 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:39.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.278+0000 7fe41c36b640 1 Processor -- start 2024-03-29T11:38:39.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.278+0000 7fe41c36b640 1 -- start start 2024-03-29T11:38:39.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.278+0000 7fe41c36b640 10 monclient: init 2024-03-29T11:38:39.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41c36b640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:39.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41c36b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:39.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41c36b640 10 monclient(hunting): picked mon.noname-a con 0x7fe4141457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41c36b640 10 monclient(hunting): picked mon.noname-c con 0x7fe41413f650 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41c36b640 10 monclient(hunting): picked mon.noname-b con 0x7fe4140a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41c36b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4140aaef0 con 0x7fe4141457f0 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41c36b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe414145bd0 con 0x7fe41413f650 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41c36b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe414145eb0 con 0x7fe4140a9d70 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41c36b640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41c36b640 10 monclient(hunting): authenticate will time out at 1039.475262s 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41a107640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fe4141457f0 0x7fe41413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:46850/0 (socket says 172.21.15.100:46850) 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe41a107640 1 -- 172.21.15.100:0/3454573644 learned_addr learned my addr 172.21.15.100:0/3454573644 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593943711 0 0) 0x7fe4140aaef0 con 0x7fe4141457f0 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): _negotiate 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:39.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe400003520 con 0x7fe4141457f0 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334019527 0 0) 0x7fe414145bd0 con 0x7fe41413f650 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): _negotiate 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4140aaef0 con 0x7fe41413f650 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94571326 0 0) 0x7fe414145eb0 con 0x7fe4140a9d70 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): _negotiate 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.280+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe400004a80 con 0x7fe4140a9d70 2024-03-29T11:38:39.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1487137591 0 0) 0x7fe4140aaef0 con 0x7fe41413f650 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 >> v1:172.21.15.100:6790/0 conn(0x7fe4140a9d70 legacy=0x7fe41414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 >> v1:172.21.15.100:6789/0 conn(0x7fe4141457f0 legacy=0x7fe41413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: _finish_hunting 0 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 1 monclient: found mon.noname-c 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe414146340 con 0x7fe41413f650 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: _finish_auth 0 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: _check_auth_tickets 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe404001520 con 0x7fe41413f650 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe41c36b640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe41c36b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:39.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: dump: 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe41c36b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:39.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe404001680 con 0x7fe41413f650 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 1 -- 172.21.15.100:0/3454573644 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe404001a90 con 0x7fe41413f650 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe41c36b640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe41c36b640 4 set_mon_vals no callback set 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.281+0000 7fe419105640 10 monclient: dump: 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:39.287 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:39.288 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:39.288 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:39.288 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:39.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.282+0000 7fe41c36b640 10 monclient: shutdown 2024-03-29T11:38:39.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.282+0000 7fe41c36b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:39.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.282+0000 7fe41c36b640 1 -- 172.21.15.100:0/3454573644 >> v1:172.21.15.191:6789/0 conn(0x7fe41413f650 legacy=0x7fe4140b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:39.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.283+0000 7fe41c36b640 1 -- 172.21.15.100:0/3454573644 shutdown_connections 2024-03-29T11:38:39.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.283+0000 7fe41c36b640 1 -- 172.21.15.100:0/3454573644 >> 172.21.15.100:0/3454573644 conn(0x7fe414066a00 msgr2=0x7fe414066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:39.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.283+0000 7fe41c36b640 1 -- 172.21.15.100:0/3454573644 shutdown_connections 2024-03-29T11:38:39.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.283+0000 7fe41c36b640 1 -- 172.21.15.100:0/3454573644 wait complete. 2024-03-29T11:38:39.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.284+0000 7fe41c36b640 10 monclient: build_initial_monmap 2024-03-29T11:38:39.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.284+0000 7fe41c36b640 10 monclient: monmap: 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.285+0000 7fe41c36b640 1 librados: starting msgr at 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.285+0000 7fe41c36b640 1 librados: starting objecter 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.285+0000 7fe41c36b640 1 Processor -- start 2024-03-29T11:38:39.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.285+0000 7fe41c36b640 1 -- start start 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.285+0000 7fe41c36b640 1 librados: setting wanted keys 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.285+0000 7fe41c36b640 1 librados: calling monclient init 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.285+0000 7fe41c36b640 10 monclient: init 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe41c36b640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe41c36b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe41c36b640 10 monclient(hunting): picked mon.noname-b con 0x7fe4140a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe41c36b640 10 monclient(hunting): picked mon.noname-a con 0x7fe41413f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe41c36b640 10 monclient(hunting): picked mon.noname-c con 0x7fe4141457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe41c36b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe414138d00 con 0x7fe41413f650 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe41c36b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe414141bd0 con 0x7fe4140a9d70 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe41c36b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe414141eb0 con 0x7fe4141457f0 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe41c36b640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe41c36b640 10 monclient(hunting): authenticate will time out at 1039.481295s 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe419906640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fe41413f650 0x7fe4141422d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:46902/0 (socket says 172.21.15.100:46902) 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe419906640 1 -- 172.21.15.100:0/68403285 learned_addr learned my addr 172.21.15.100:0/68403285 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:39.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe40affd640 10 client.?.objecter ms_handle_connect 0x7fe41413f650 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe40affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811174562 0 0) 0x7fe414138d00 con 0x7fe41413f650 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe40affd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe40affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe40affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe40affd640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f8004eb0 con 0x7fe41413f650 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe40affd640 10 client.?.objecter ms_handle_connect 0x7fe4141457f0 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.286+0000 7fe40affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.287+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 42249465 0 0) 0x7fe3f8004eb0 con 0x7fe41413f650 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.287+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe414138d00 con 0x7fe41413f650 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.287+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe404001a50 con 0x7fe41413f650 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.287+0000 7fe40affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:39.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.287+0000 7fe40affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.287+0000 7fe40affd640 10 monclient(hunting): dump: 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.287+0000 7fe40affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.287+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2240206733 0 0) 0x7fe414138d00 con 0x7fe41413f650 2024-03-29T11:38:39.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.287+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 >> v1:172.21.15.100:6790/0 conn(0x7fe4141457f0 legacy=0x7fe4141e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.287+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 >> v1:172.21.15.191:6789/0 conn(0x7fe4140a9d70 legacy=0x7fe4141352f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe40affd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe40affd640 1 monclient: found mon.a 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe40affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe40affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41413b6f0 con 0x7fe41413f650 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe40affd640 10 monclient: _finish_auth 0 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe40affd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe40affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe41c36b640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe41c36b640 10 monclient: _renew_subs 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe41c36b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:38:39.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.289+0000 7fe41c36b640 1 -- 172.21.15.100:0/68403285 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe414140c90 con 0x7fe41413f650 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe41c36b640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe41c36b640 10 monclient: _renew_subs 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe41c36b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe41c36b640 1 -- 172.21.15.100:0/68403285 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41413fda0 con 0x7fe41413f650 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe41c36b640 1 librados: init done 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4040038a0 con 0x7fe41413f650 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe40affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe418904640 4 set_mon_vals no callback set 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe404003cb0 con 0x7fe41413f650 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe40affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.290+0000 7fe40affd640 10 monclient: dump: 2024-03-29T11:38:39.300 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.292+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fe404003e10 con 0x7fe41413f650 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.292+0000 7fe41c36b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.292+0000 7fe41c36b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.292+0000 7fe41c36b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:38:39.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.292+0000 7fe41c36b640 1 -- 172.21.15.100:0/68403285 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3fc005080 con 0x7fe41413f650 2024-03-29T11:38:39.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.292+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fe404084bb0 con 0x7fe41413f650 2024-03-29T11:38:39.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.292+0000 7fe40affd640 10 client.4314.objecter ms_dispatch 0x7fe414000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:39.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.292+0000 7fe40affd640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:39.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.292+0000 7fe40affd640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:39.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.292+0000 7fe40affd640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-29T11:38:39.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.293+0000 7fe41a107640 10 monclient: get_auth_request con 0x7fe3f806dce0 auth_method 0 2024-03-29T11:38:39.302 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.293+0000 7fe40affd640 10 client.4314.objecter ms_handle_connect 0x7fe3f806dce0 2024-03-29T11:38:39.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.301+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe404050490 con 0x7fe41413f650 2024-03-29T11:38:39.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.301+0000 7fe40affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:39.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.301+0000 7fe40affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:39.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.615+0000 7fe41c36b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:39.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.615+0000 7fe41c36b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:39.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.615+0000 7fe41c36b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:38:39.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.615+0000 7fe41c36b640 1 -- 172.21.15.100:0/68403285 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3fc005330 con 0x7fe41413f650 2024-03-29T11:38:39.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.616+0000 7fe40affd640 1 -- 172.21.15.100:0/68403285 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fe4040523c0 con 0x7fe41413f650 2024-03-29T11:38:39.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.616+0000 7fe40affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:39.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.616+0000 7fe40affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:39.617 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:38:39.617 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:39.621 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 10 librados: watch_flush enter 2024-03-29T11:38:39.622 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 10 librados: watch_flush exit 2024-03-29T11:38:39.622 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-29T11:38:39.622 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 10 client.4314.objecter successfully canceled tick 2024-03-29T11:38:39.622 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 1 -- 172.21.15.100:0/68403285 >> v1:172.21.15.100:6816/3220289657 conn(0x7fe3f806dce0 legacy=0x7fe3f80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:39.622 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 10 monclient: shutdown 2024-03-29T11:38:39.622 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:39.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 1 -- 172.21.15.100:0/68403285 >> v1:172.21.15.100:6789/0 conn(0x7fe41413f650 legacy=0x7fe4141422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:39.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 1 -- 172.21.15.100:0/68403285 shutdown_connections 2024-03-29T11:38:39.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 1 -- 172.21.15.100:0/68403285 >> 172.21.15.100:0/68403285 conn(0x7fe414066a00 msgr2=0x7fe414066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:39.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 1 -- 172.21.15.100:0/68403285 shutdown_connections 2024-03-29T11:38:39.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 1 -- 172.21.15.100:0/68403285 wait complete. 2024-03-29T11:38:39.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:39.619+0000 7fe41c36b640 1 librados: shutdown 2024-03-29T11:38:39.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:46.441 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:46.593 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.592+0000 7fe62037b640 10 monclient: get_monmap_and_config 2024-03-29T11:38:46.593 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.592+0000 7fe62037b640 10 monclient: build_initial_monmap 2024-03-29T11:38:46.593 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.592+0000 7fe62037b640 10 monclient: monmap: 2024-03-29T11:38:46.593 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:46.593 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:46.593 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:38:46.593817+0000 2024-03-29T11:38:46.594 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:38:46.593817+0000 2024-03-29T11:38:46.594 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:46.594 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:46.594 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:46.594 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:38:46.594 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:38:46.594 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:46.594 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.593+0000 7fe62037b640 1 Processor -- start 2024-03-29T11:38:46.594 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.593+0000 7fe62037b640 1 -- start start 2024-03-29T11:38:46.594 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.593+0000 7fe62037b640 10 monclient: init 2024-03-29T11:38:46.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.594+0000 7fe62037b640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:46.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.594+0000 7fe62037b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:46.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.594+0000 7fe62037b640 10 monclient(hunting): picked mon.noname-c con 0x7fe6181422b0 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:46.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.594+0000 7fe62037b640 10 monclient(hunting): picked mon.noname-a con 0x7fe618142690 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:46.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.594+0000 7fe62037b640 10 monclient(hunting): picked mon.noname-b con 0x7fe6180b17d0 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:46.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.594+0000 7fe62037b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe618142c80 con 0x7fe618142690 2024-03-29T11:38:46.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.594+0000 7fe62037b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180aae40 con 0x7fe6181422b0 2024-03-29T11:38:46.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.594+0000 7fe62037b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61813fbd0 con 0x7fe6180b17d0 2024-03-29T11:38:46.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.594+0000 7fe62037b640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:46.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.594+0000 7fe62037b640 10 monclient(hunting): authenticate will time out at 1046.790047s 2024-03-29T11:38:46.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.595+0000 7fe61e117640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe6181422b0 0x7fe61813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:47926/0 (socket says 172.21.15.100:47926) 2024-03-29T11:38:46.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.595+0000 7fe61e117640 1 -- 172.21.15.100:0/1859173497 learned_addr learned my addr 172.21.15.100:0/1859173497 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:46.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.595+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638901327 0 0) 0x7fe6180aae40 con 0x7fe6181422b0 2024-03-29T11:38:46.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.595+0000 7fe61d115640 10 monclient(hunting): _negotiate 2024-03-29T11:38:46.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.595+0000 7fe61d115640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:46.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.595+0000 7fe61d115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:46.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.595+0000 7fe61d115640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:38:46.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.595+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe604003520 con 0x7fe6181422b0 2024-03-29T11:38:46.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.595+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335963357 0 0) 0x7fe61813fbd0 con 0x7fe6180b17d0 2024-03-29T11:38:46.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient(hunting): _negotiate 2024-03-29T11:38:46.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6180aae40 con 0x7fe6180b17d0 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4023403324 0 0) 0x7fe604003520 con 0x7fe6181422b0 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 >> v1:172.21.15.100:6790/0 conn(0x7fe6180b17d0 legacy=0x7fe6180b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 >> v1:172.21.15.100:6789/0 conn(0x7fe618142690 legacy=0x7fe6180b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: _finish_hunting 0 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 1 monclient: found mon.noname-c 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe618151940 con 0x7fe6181422b0 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: _finish_auth 0 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: _check_auth_tickets 2024-03-29T11:38:46.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe62037b640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe62037b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe614001690 con 0x7fe6181422b0 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: dump: 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:46.608 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe62037b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6140018b0 con 0x7fe6181422b0 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe62037b640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe62037b640 4 set_mon_vals no callback set 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 1 -- 172.21.15.100:0/1859173497 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe614001bc0 con 0x7fe6181422b0 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.596+0000 7fe61d115640 10 monclient: dump: 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:46.609 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.597+0000 7fe62037b640 10 monclient: shutdown 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.597+0000 7fe62037b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.597+0000 7fe62037b640 1 -- 172.21.15.100:0/1859173497 >> v1:172.21.15.191:6789/0 conn(0x7fe6181422b0 legacy=0x7fe61813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.597+0000 7fe62037b640 1 -- 172.21.15.100:0/1859173497 shutdown_connections 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.597+0000 7fe62037b640 1 -- 172.21.15.100:0/1859173497 >> 172.21.15.100:0/1859173497 conn(0x7fe618066a00 msgr2=0x7fe618066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.597+0000 7fe62037b640 1 -- 172.21.15.100:0/1859173497 shutdown_connections 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.597+0000 7fe62037b640 1 -- 172.21.15.100:0/1859173497 wait complete. 2024-03-29T11:38:46.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.598+0000 7fe62037b640 10 monclient: build_initial_monmap 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.598+0000 7fe62037b640 10 monclient: monmap: 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.599+0000 7fe62037b640 1 librados: starting msgr at 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.599+0000 7fe62037b640 1 librados: starting objecter 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.599+0000 7fe62037b640 1 Processor -- start 2024-03-29T11:38:46.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.599+0000 7fe62037b640 1 -- start start 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.599+0000 7fe62037b640 1 librados: setting wanted keys 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.599+0000 7fe62037b640 1 librados: calling monclient init 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.599+0000 7fe62037b640 10 monclient: init 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe62037b640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe62037b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe62037b640 10 monclient(hunting): picked mon.noname-a con 0x7fe6180b17d0 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe62037b640 10 monclient(hunting): picked mon.noname-c con 0x7fe6181422b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe62037b640 10 monclient(hunting): picked mon.noname-b con 0x7fe618142690 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe62037b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe618132200 con 0x7fe6180b17d0 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe62037b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe618131dd0 con 0x7fe618142690 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe62037b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61812ba40 con 0x7fe6181422b0 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe62037b640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe62037b640 10 monclient(hunting): authenticate will time out at 1046.795594s 2024-03-29T11:38:46.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe61d916640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fe6181422b0 0x7fe6181dbb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:37154/0 (socket says 172.21.15.100:37154) 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe61d916640 1 -- 172.21.15.100:0/3663480401 learned_addr learned my addr 172.21.15.100:0/3663480401 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe60effd640 10 client.?.objecter ms_handle_connect 0x7fe6181422b0 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.600+0000 7fe60effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 client.?.objecter ms_handle_connect 0x7fe6180b17d0 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 client.?.objecter ms_handle_connect 0x7fe618142690 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444940895 0 0) 0x7fe61812ba40 con 0x7fe6181422b0 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5fc004eb0 con 0x7fe6181422b0 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37684332 0 0) 0x7fe618131dd0 con 0x7fe618142690 2024-03-29T11:38:46.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe61812ba40 con 0x7fe618142690 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082098267 0 0) 0x7fe618132200 con 0x7fe6180b17d0 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe618131dd0 con 0x7fe6180b17d0 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3481961161 0 0) 0x7fe5fc004eb0 con 0x7fe6181422b0 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5fc006680 con 0x7fe6181422b0 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.601+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe608001890 con 0x7fe6181422b0 2024-03-29T11:38:46.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 10 monclient(hunting): dump: 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:46.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2909872673 0 0) 0x7fe618131dd0 con 0x7fe6180b17d0 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5fc004eb0 con 0x7fe6180b17d0 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 134296264 0 0) 0x7fe5fc006680 con 0x7fe6181422b0 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 >> v1:172.21.15.191:6789/0 conn(0x7fe618142690 legacy=0x7fe6181e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 >> v1:172.21.15.100:6789/0 conn(0x7fe6180b17d0 legacy=0x7fe61812ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 1 monclient: found mon.c 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61812bbd0 con 0x7fe6181422b0 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 10 monclient: _finish_auth 0 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe60effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe62037b640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:38:46.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe62037b640 10 monclient: _renew_subs 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe62037b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe62037b640 1 -- 172.21.15.100:0/3663480401 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6181312c0 con 0x7fe6181422b0 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe62037b640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe62037b640 10 monclient: _renew_subs 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe62037b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe62037b640 1 -- 172.21.15.100:0/3663480401 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe618150720 con 0x7fe6181422b0 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.602+0000 7fe62037b640 1 librados: init done 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.603+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe608003610 con 0x7fe6181422b0 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.603+0000 7fe60effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.603+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe608001d80 con 0x7fe6181422b0 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.603+0000 7fe60effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.603+0000 7fe60effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.603+0000 7fe60effd640 10 monclient: dump: 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:46.617 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.603+0000 7fe61c914640 4 set_mon_vals no callback set 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.604+0000 7fe62037b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.604+0000 7fe62037b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.604+0000 7fe62037b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:46.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.604+0000 7fe62037b640 1 -- 172.21.15.100:0/3663480401 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe600005080 con 0x7fe6181422b0 2024-03-29T11:38:46.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.606+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fe6080039e0 con 0x7fe6181422b0 2024-03-29T11:38:46.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.607+0000 7fe61e117640 10 monclient: get_auth_request con 0x7fe5fc06e050 auth_method 0 2024-03-29T11:38:46.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.607+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fe608084940 con 0x7fe6181422b0 2024-03-29T11:38:46.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.607+0000 7fe60effd640 10 client.4316.objecter ms_dispatch 0x7fe618000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:46.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.607+0000 7fe60effd640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:46.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.607+0000 7fe60effd640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:46.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.607+0000 7fe60effd640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T11:38:46.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.607+0000 7fe60effd640 10 client.4316.objecter ms_handle_connect 0x7fe5fc06e050 2024-03-29T11:38:46.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.617+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe608050220 con 0x7fe6181422b0 2024-03-29T11:38:46.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.617+0000 7fe60effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:46.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.617+0000 7fe60effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:46.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.985+0000 7fe62037b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:46.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.985+0000 7fe62037b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:46.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.985+0000 7fe62037b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:46.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.985+0000 7fe62037b640 1 -- 172.21.15.100:0/3663480401 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe600005330 con 0x7fe6181422b0 2024-03-29T11:38:46.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.986+0000 7fe60effd640 1 -- 172.21.15.100:0/3663480401 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fe608052150 con 0x7fe6181422b0 2024-03-29T11:38:46.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.986+0000 7fe60effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:46.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.986+0000 7fe60effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:46.987 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:38:46.987 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:46.990 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.988+0000 7fe62037b640 10 librados: watch_flush enter 2024-03-29T11:38:46.990 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 10 librados: watch_flush exit 2024-03-29T11:38:46.990 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T11:38:46.990 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 10 client.4316.objecter successfully canceled tick 2024-03-29T11:38:46.990 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 1 -- 172.21.15.100:0/3663480401 >> v1:172.21.15.100:6816/3220289657 conn(0x7fe5fc06e050 legacy=0x7fe5fc070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:46.990 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 10 monclient: shutdown 2024-03-29T11:38:46.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:46.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 1 -- 172.21.15.100:0/3663480401 >> v1:172.21.15.100:6790/0 conn(0x7fe6181422b0 legacy=0x7fe6181dbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:46.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 1 -- 172.21.15.100:0/3663480401 shutdown_connections 2024-03-29T11:38:46.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 1 -- 172.21.15.100:0/3663480401 >> 172.21.15.100:0/3663480401 conn(0x7fe618066a00 msgr2=0x7fe61813ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:46.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 1 -- 172.21.15.100:0/3663480401 shutdown_connections 2024-03-29T11:38:46.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 1 -- 172.21.15.100:0/3663480401 wait complete. 2024-03-29T11:38:46.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:46.989+0000 7fe62037b640 1 librados: shutdown 2024-03-29T11:38:47.003 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:53.810 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:53.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.956+0000 7f5d36582640 10 monclient: get_monmap_and_config 2024-03-29T11:38:53.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.956+0000 7f5d36582640 10 monclient: build_initial_monmap 2024-03-29T11:38:53.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.956+0000 7f5d36582640 10 monclient: monmap: 2024-03-29T11:38:53.957 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:53.957 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:53.957 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:38:53.957579+0000 2024-03-29T11:38:53.957 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:38:53.957579+0000 2024-03-29T11:38:53.957 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:53.958 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:53.958 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:53.958 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:38:53.958 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:38:53.958 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:53.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.958+0000 7f5d36582640 1 Processor -- start 2024-03-29T11:38:53.960 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.959+0000 7f5d36582640 1 -- start start 2024-03-29T11:38:53.960 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.959+0000 7f5d36582640 10 monclient: init 2024-03-29T11:38:53.960 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.959+0000 7f5d36582640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:53.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.959+0000 7f5d36582640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:53.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.960+0000 7f5d36582640 10 monclient(hunting): picked mon.noname-a con 0x7f5d301457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:53.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.960+0000 7f5d36582640 10 monclient(hunting): picked mon.noname-b con 0x7f5d3013f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:53.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.960+0000 7f5d36582640 10 monclient(hunting): picked mon.noname-c con 0x7f5d300a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:53.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.960+0000 7f5d36582640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d300aaef0 con 0x7f5d301457f0 2024-03-29T11:38:53.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.960+0000 7f5d36582640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d30145bd0 con 0x7f5d300a9d70 2024-03-29T11:38:53.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.960+0000 7f5d36582640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d30145eb0 con 0x7f5d3013f650 2024-03-29T11:38:53.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.960+0000 7f5d36582640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:53.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.960+0000 7f5d36582640 10 monclient(hunting): authenticate will time out at 1054.155159s 2024-03-29T11:38:53.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.960+0000 7f5d2f7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5d3013f650 0x7f5d300b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:47896/0 (socket says 172.21.15.100:47896) 2024-03-29T11:38:53.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.960+0000 7f5d2f7fe640 1 -- 172.21.15.100:0/2160393979 learned_addr learned my addr 172.21.15.100:0/2160393979 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:53.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.963+0000 7f5d2effd640 1 -- 172.21.15.100:0/2160393979 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249875495 0 0) 0x7f5d30145eb0 con 0x7f5d3013f650 2024-03-29T11:38:53.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.963+0000 7f5d2effd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:53.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.963+0000 7f5d2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:53.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.963+0000 7f5d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:53.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.963+0000 7f5d2effd640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:38:53.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.963+0000 7f5d2effd640 1 -- 172.21.15.100:0/2160393979 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d18003520 con 0x7f5d3013f650 2024-03-29T11:38:53.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.963+0000 7f5d2effd640 1 -- 172.21.15.100:0/2160393979 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4186117445 0 0) 0x7f5d18003520 con 0x7f5d3013f650 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.963+0000 7f5d2effd640 1 -- 172.21.15.100:0/2160393979 >> v1:172.21.15.191:6789/0 conn(0x7f5d300a9d70 legacy=0x7f5d3014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 1 -- 172.21.15.100:0/2160393979 >> v1:172.21.15.100:6789/0 conn(0x7f5d301457f0 legacy=0x7f5d3013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 1 monclient: found mon.noname-b 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 1 -- 172.21.15.100:0/2160393979 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d30146340 con 0x7f5d3013f650 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: _finish_auth 0 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 1 -- 172.21.15.100:0/2160393979 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5d20001690 con 0x7f5d3013f650 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: dump: 2024-03-29T11:38:53.965 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:53.966 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:53.966 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:53.966 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:53.966 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:53.966 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:53.966 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:53.966 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:53.966 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:53.966 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d36582640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d36582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 1 -- 172.21.15.100:0/2160393979 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d200017f0 con 0x7f5d3013f650 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 1 -- 172.21.15.100:0/2160393979 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5d20001c40 con 0x7f5d3013f650 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d36582640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d36582640 4 set_mon_vals no callback set 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.964+0000 7f5d2effd640 10 monclient: dump: 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:53.968 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.965+0000 7f5d36582640 10 monclient: shutdown 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.965+0000 7f5d36582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.965+0000 7f5d36582640 1 -- 172.21.15.100:0/2160393979 >> v1:172.21.15.100:6790/0 conn(0x7f5d3013f650 legacy=0x7f5d300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.965+0000 7f5d36582640 1 -- 172.21.15.100:0/2160393979 shutdown_connections 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.965+0000 7f5d36582640 1 -- 172.21.15.100:0/2160393979 >> 172.21.15.100:0/2160393979 conn(0x7f5d30066a00 msgr2=0x7f5d30066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:53.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.966+0000 7f5d36582640 1 -- 172.21.15.100:0/2160393979 shutdown_connections 2024-03-29T11:38:53.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.966+0000 7f5d36582640 1 -- 172.21.15.100:0/2160393979 wait complete. 2024-03-29T11:38:53.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.967+0000 7f5d36582640 10 monclient: build_initial_monmap 2024-03-29T11:38:53.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.967+0000 7f5d36582640 10 monclient: monmap: 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.968+0000 7f5d36582640 1 librados: starting msgr at 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.968+0000 7f5d36582640 1 librados: starting objecter 2024-03-29T11:38:53.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.968+0000 7f5d36582640 1 Processor -- start 2024-03-29T11:38:53.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.968+0000 7f5d36582640 1 -- start start 2024-03-29T11:38:53.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.968+0000 7f5d36582640 1 librados: setting wanted keys 2024-03-29T11:38:53.974 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.968+0000 7f5d36582640 1 librados: calling monclient init 2024-03-29T11:38:53.974 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.968+0000 7f5d36582640 10 monclient: init 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.969+0000 7f5d36582640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.969+0000 7f5d36582640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.969+0000 7f5d36582640 10 monclient(hunting): picked mon.noname-b con 0x7f5d300a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.969+0000 7f5d36582640 10 monclient(hunting): picked mon.noname-c con 0x7f5d301457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.969+0000 7f5d36582640 10 monclient(hunting): picked mon.noname-a con 0x7f5d3006de90 addr v1:172.21.15.100:6789/0 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.969+0000 7f5d36582640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3006f5d0 con 0x7f5d3006de90 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.969+0000 7f5d36582640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d30071d60 con 0x7f5d300a9d70 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.969+0000 7f5d36582640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d301e7690 con 0x7f5d301457f0 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.969+0000 7f5d36582640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.969+0000 7f5d36582640 10 monclient(hunting): authenticate will time out at 1054.164814s 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.971+0000 7f5d2f7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5d301457f0 0x7f5d301e14a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:48328/0 (socket says 172.21.15.100:48328) 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.971+0000 7f5d2f7fe640 1 -- 172.21.15.100:0/2799650556 learned_addr learned my addr 172.21.15.100:0/2799650556 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.973+0000 7f5d2cff9640 10 client.?.objecter ms_handle_connect 0x7f5d301457f0 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.973+0000 7f5d2cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:53.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.973+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089747647 0 0) 0x7f5d301e7690 con 0x7f5d301457f0 2024-03-29T11:38:53.976 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.973+0000 7f5d2cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:53.976 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.973+0000 7f5d2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:53.976 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.973+0000 7f5d2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:53.976 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.973+0000 7f5d2cff9640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:38:53.976 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.973+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d1c004eb0 con 0x7f5d301457f0 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.974+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3677623085 0 0) 0x7f5d1c004eb0 con 0x7f5d301457f0 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.974+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d301e7690 con 0x7f5d301457f0 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 10 client.?.objecter ms_handle_connect 0x7f5d3006de90 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5d20001b00 con 0x7f5d301457f0 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 10 monclient(hunting): dump: 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:53.977 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1294098995 0 0) 0x7f5d301e7690 con 0x7f5d301457f0 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 >> v1:172.21.15.191:6789/0 conn(0x7f5d300a9d70 legacy=0x7f5d301425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 >> v1:172.21.15.100:6789/0 conn(0x7f5d3006de90 legacy=0x7f5d301e7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 1 monclient: found mon.c 2024-03-29T11:38:53.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3006f840 con 0x7f5d301457f0 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d36582640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d36582640 10 monclient: _renew_subs 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d36582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d36582640 1 -- 172.21.15.100:0/2799650556 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3013fda0 con 0x7f5d301457f0 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d36582640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d36582640 10 monclient: _renew_subs 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d36582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d36582640 1 -- 172.21.15.100:0/2799650556 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d301e1bb0 con 0x7f5d301457f0 2024-03-29T11:38:53.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.975+0000 7f5d36582640 1 librados: init done 2024-03-29T11:38:53.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.977+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d20004bf0 con 0x7f5d301457f0 2024-03-29T11:38:53.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.977+0000 7f5d2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:53.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.977+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5d20004f80 con 0x7f5d301457f0 2024-03-29T11:38:53.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.977+0000 7f5d2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.977+0000 7f5d2cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.977+0000 7f5d2cff9640 10 monclient: dump: 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.977+0000 7f5d2e7fc640 4 set_mon_vals no callback set 2024-03-29T11:38:53.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.978+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f5d200051a0 con 0x7f5d301457f0 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.978+0000 7f5d36582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.978+0000 7f5d36582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.978+0000 7f5d36582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.978+0000 7f5d36582640 1 -- 172.21.15.100:0/2799650556 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d10005080 con 0x7f5d301457f0 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.979+0000 7f5d2ffff640 10 monclient: get_auth_request con 0x7f5d1c06dc10 auth_method 0 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.980+0000 7f5d2cff9640 10 client.4322.objecter ms_handle_connect 0x7f5d1c06dc10 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.980+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f5d20084c30 con 0x7f5d301457f0 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.980+0000 7f5d2cff9640 10 client.4322.objecter ms_dispatch 0x7f5d30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.980+0000 7f5d2cff9640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.980+0000 7f5d2cff9640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:53.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.981+0000 7f5d2cff9640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T11:38:53.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.987+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d20050510 con 0x7f5d301457f0 2024-03-29T11:38:53.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.987+0000 7f5d2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:53.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:53.987+0000 7f5d2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:54.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.314+0000 7f5d36582640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:54.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.314+0000 7f5d36582640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:54.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.314+0000 7f5d36582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:38:54.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.314+0000 7f5d36582640 1 -- 172.21.15.100:0/2799650556 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d10005330 con 0x7f5d301457f0 2024-03-29T11:38:54.317 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.316+0000 7f5d2cff9640 1 -- 172.21.15.100:0/2799650556 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f5d20052440 con 0x7f5d301457f0 2024-03-29T11:38:54.317 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.316+0000 7f5d2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:54.317 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.316+0000 7f5d2cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:54.317 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:38:54.318 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:54.321 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.320+0000 7f5d36582640 10 librados: watch_flush enter 2024-03-29T11:38:54.322 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.320+0000 7f5d36582640 10 librados: watch_flush exit 2024-03-29T11:38:54.322 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.320+0000 7f5d36582640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T11:38:54.322 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.320+0000 7f5d36582640 10 client.4322.objecter successfully canceled tick 2024-03-29T11:38:54.322 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.321+0000 7f5d36582640 1 -- 172.21.15.100:0/2799650556 >> v1:172.21.15.100:6816/3220289657 conn(0x7f5d1c06dc10 legacy=0x7f5d1c0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:54.322 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.321+0000 7f5d36582640 10 monclient: shutdown 2024-03-29T11:38:54.322 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.321+0000 7f5d36582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:54.322 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.321+0000 7f5d36582640 1 -- 172.21.15.100:0/2799650556 >> v1:172.21.15.100:6790/0 conn(0x7f5d301457f0 legacy=0x7f5d301e14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:54.322 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.321+0000 7f5d36582640 1 -- 172.21.15.100:0/2799650556 shutdown_connections 2024-03-29T11:38:54.322 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.321+0000 7f5d36582640 1 -- 172.21.15.100:0/2799650556 >> 172.21.15.100:0/2799650556 conn(0x7f5d30066a00 msgr2=0x7f5d300b4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:54.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.321+0000 7f5d36582640 1 -- 172.21.15.100:0/2799650556 shutdown_connections 2024-03-29T11:38:54.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.321+0000 7f5d36582640 1 -- 172.21.15.100:0/2799650556 wait complete. 2024-03-29T11:38:54.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:38:54.321+0000 7f5d36582640 1 librados: shutdown 2024-03-29T11:38:54.339 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:01.145 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:01.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.291+0000 7fc4657bc640 10 monclient: get_monmap_and_config 2024-03-29T11:39:01.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.291+0000 7fc4657bc640 10 monclient: build_initial_monmap 2024-03-29T11:39:01.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.291+0000 7fc4657bc640 10 monclient: monmap: 2024-03-29T11:39:01.293 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:01.293 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:01.293 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:39:01.292600+0000 2024-03-29T11:39:01.293 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:39:01.292600+0000 2024-03-29T11:39:01.293 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:01.294 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:01.294 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:01.294 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:39:01.294 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:39:01.294 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:01.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.292+0000 7fc4657bc640 1 Processor -- start 2024-03-29T11:39:01.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.292+0000 7fc4657bc640 1 -- start start 2024-03-29T11:39:01.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.292+0000 7fc4657bc640 10 monclient: init 2024-03-29T11:39:01.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.293+0000 7fc4657bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:01.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.293+0000 7fc4657bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:01.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.293+0000 7fc4657bc640 10 monclient(hunting): picked mon.noname-b con 0x7fc4600a9930 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:01.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.293+0000 7fc4657bc640 10 monclient(hunting): picked mon.noname-a con 0x7fc460149dc0 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:01.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.293+0000 7fc4657bc640 10 monclient(hunting): picked mon.noname-c con 0x7fc46014ff40 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:01.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.293+0000 7fc4657bc640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4600aae60 con 0x7fc460149dc0 2024-03-29T11:39:01.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.293+0000 7fc4657bc640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4600b4310 con 0x7fc46014ff40 2024-03-29T11:39:01.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.293+0000 7fc4657bc640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4600afdd0 con 0x7fc4600a9930 2024-03-29T11:39:01.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.293+0000 7fc4657bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:01.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.293+0000 7fc4657bc640 10 monclient(hunting): authenticate will time out at 1061.488719s 2024-03-29T11:39:01.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.295+0000 7fc45effd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fc4600a9930 0x7fc460143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:41496/0 (socket says 172.21.15.100:41496) 2024-03-29T11:39:01.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.295+0000 7fc45effd640 1 -- 172.21.15.100:0/886638892 learned_addr learned my addr 172.21.15.100:0/886638892 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:01.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.297+0000 7fc45dffb640 1 -- 172.21.15.100:0/886638892 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315180316 0 0) 0x7fc4600afdd0 con 0x7fc4600a9930 2024-03-29T11:39:01.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.297+0000 7fc45dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:01.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.297+0000 7fc45dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:01.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.297+0000 7fc45dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:01.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.297+0000 7fc45dffb640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:39:01.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.297+0000 7fc45dffb640 1 -- 172.21.15.100:0/886638892 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc448003520 con 0x7fc4600a9930 2024-03-29T11:39:01.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 1 -- 172.21.15.100:0/886638892 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1849159451 0 0) 0x7fc448003520 con 0x7fc4600a9930 2024-03-29T11:39:01.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 1 -- 172.21.15.100:0/886638892 >> v1:172.21.15.191:6789/0 conn(0x7fc46014ff40 legacy=0x7fc460152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 1 -- 172.21.15.100:0/886638892 >> v1:172.21.15.100:6789/0 conn(0x7fc460149dc0 legacy=0x7fc46014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 1 monclient: found mon.noname-b 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 1 -- 172.21.15.100:0/886638892 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46014a950 con 0x7fc4600a9930 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 1 -- 172.21.15.100:0/886638892 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc454001690 con 0x7fc4600a9930 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: dump: 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:01.300 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 1 -- 172.21.15.100:0/886638892 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc454001920 con 0x7fc4600a9930 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 1 -- 172.21.15.100:0/886638892 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc454001cb0 con 0x7fc4600a9930 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.298+0000 7fc45dffb640 10 monclient: dump: 2024-03-29T11:39:01.301 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:01.302 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:01.302 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:01.302 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:01.302 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:01.302 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:01.302 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:01.302 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:01.302 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:01.302 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:01.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.299+0000 7fc4657bc640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:39:01.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.299+0000 7fc4657bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:01.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.299+0000 7fc4657bc640 4 set_mon_vals no callback set 2024-03-29T11:39:01.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.300+0000 7fc4657bc640 10 monclient: shutdown 2024-03-29T11:39:01.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.300+0000 7fc4657bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:01.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.300+0000 7fc4657bc640 1 -- 172.21.15.100:0/886638892 >> v1:172.21.15.100:6790/0 conn(0x7fc4600a9930 legacy=0x7fc460143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:01.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.300+0000 7fc4657bc640 1 -- 172.21.15.100:0/886638892 shutdown_connections 2024-03-29T11:39:01.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.300+0000 7fc4657bc640 1 -- 172.21.15.100:0/886638892 >> 172.21.15.100:0/886638892 conn(0x7fc4600b35e0 msgr2=0x7fc4600b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:01.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.303+0000 7fc4657bc640 1 -- 172.21.15.100:0/886638892 shutdown_connections 2024-03-29T11:39:01.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.304+0000 7fc4657bc640 1 -- 172.21.15.100:0/886638892 wait complete. 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.304+0000 7fc4657bc640 10 monclient: build_initial_monmap 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.304+0000 7fc4657bc640 10 monclient: monmap: 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:01.307 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.305+0000 7fc4657bc640 1 librados: starting msgr at 2024-03-29T11:39:01.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.305+0000 7fc4657bc640 1 librados: starting objecter 2024-03-29T11:39:01.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.305+0000 7fc4657bc640 1 Processor -- start 2024-03-29T11:39:01.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.305+0000 7fc4657bc640 1 -- start start 2024-03-29T11:39:01.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.305+0000 7fc4657bc640 1 librados: setting wanted keys 2024-03-29T11:39:01.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.305+0000 7fc4657bc640 1 librados: calling monclient init 2024-03-29T11:39:01.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.305+0000 7fc4657bc640 10 monclient: init 2024-03-29T11:39:01.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.306+0000 7fc4657bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:01.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.306+0000 7fc4657bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.306+0000 7fc4657bc640 10 monclient(hunting): picked mon.noname-a con 0x7fc460149dc0 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.306+0000 7fc4657bc640 10 monclient(hunting): picked mon.noname-c con 0x7fc46014ff40 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.306+0000 7fc4657bc640 10 monclient(hunting): picked mon.noname-b con 0x7fc4601dc570 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.306+0000 7fc4657bc640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4601db980 con 0x7fc460149dc0 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.306+0000 7fc4657bc640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46013e040 con 0x7fc4601dc570 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.306+0000 7fc4657bc640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46013e1d0 con 0x7fc46014ff40 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.306+0000 7fc4657bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.306+0000 7fc4657bc640 10 monclient(hunting): authenticate will time out at 1061.501700s 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.308+0000 7fc45e7fc640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fc46014ff40 0x7fc460148d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:41804/0 (socket says 172.21.15.100:41804) 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.308+0000 7fc45e7fc640 1 -- 172.21.15.100:0/175734511 learned_addr learned my addr 172.21.15.100:0/175734511 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.308+0000 7fc4477fe640 10 client.?.objecter ms_handle_connect 0x7fc46014ff40 2024-03-29T11:39:01.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.308+0000 7fc4477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:01.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.308+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492108164 0 0) 0x7fc46013e1d0 con 0x7fc46014ff40 2024-03-29T11:39:01.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.308+0000 7fc4477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:01.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.308+0000 7fc4477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:01.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.308+0000 7fc4477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:01.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.308+0000 7fc4477fe640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:39:01.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.309+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc44c004eb0 con 0x7fc46014ff40 2024-03-29T11:39:01.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.309+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1179114424 0 0) 0x7fc44c004eb0 con 0x7fc46014ff40 2024-03-29T11:39:01.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.309+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc46013e1d0 con 0x7fc46014ff40 2024-03-29T11:39:01.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.309+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc458001ce0 con 0x7fc46014ff40 2024-03-29T11:39:01.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.309+0000 7fc4477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.309+0000 7fc4477fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.309+0000 7fc4477fe640 10 monclient(hunting): dump: 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:01.311 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.309+0000 7fc4477fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:01.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 10 client.?.objecter ms_handle_connect 0x7fc460149dc0 2024-03-29T11:39:01.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:01.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308302061 0 0) 0x7fc4601db980 con 0x7fc460149dc0 2024-03-29T11:39:01.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:01.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:01.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:01.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:39:01.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc44c004eb0 con 0x7fc460149dc0 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4015945776 0 0) 0x7fc44c004eb0 con 0x7fc460149dc0 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4601db980 con 0x7fc460149dc0 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc454001550 con 0x7fc460149dc0 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.311+0000 7fc4477fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 10 monclient(hunting): dump: 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:01.313 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:01.314 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:01.314 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:01.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3844013272 0 0) 0x7fc46013e1d0 con 0x7fc46014ff40 2024-03-29T11:39:01.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 >> v1:172.21.15.191:6789/0 conn(0x7fc4601dc570 legacy=0x7fc4601e6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:01.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 >> v1:172.21.15.100:6789/0 conn(0x7fc460149dc0 legacy=0x7fc4601dbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:01.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 1 monclient: found mon.c 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4601494a0 con 0x7fc46014ff40 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4657bc640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4657bc640 10 monclient: _renew_subs 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4657bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4657bc640 1 -- 172.21.15.100:0/175734511 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc460149710 con 0x7fc46014ff40 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4657bc640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4657bc640 10 monclient: _renew_subs 2024-03-29T11:39:01.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4657bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:01.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4657bc640 1 -- 172.21.15.100:0/175734511 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc460148340 con 0x7fc46014ff40 2024-03-29T11:39:01.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.312+0000 7fc4657bc640 1 librados: init done 2024-03-29T11:39:01.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.313+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4580020a0 con 0x7fc46014ff40 2024-03-29T11:39:01.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.313+0000 7fc4477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:01.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.313+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc458003de0 con 0x7fc46014ff40 2024-03-29T11:39:01.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.313+0000 7fc45cff9640 4 set_mon_vals no callback set 2024-03-29T11:39:01.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.314+0000 7fc4657bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:01.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.316+0000 7fc4657bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:01.318 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.316+0000 7fc4657bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:01.318 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.316+0000 7fc4657bc640 1 -- 172.21.15.100:0/175734511 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc440005080 con 0x7fc46014ff40 2024-03-29T11:39:01.318 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.317+0000 7fc4477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:01.318 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.317+0000 7fc4477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:01.318 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.317+0000 7fc4477fe640 10 monclient: dump: 2024-03-29T11:39:01.318 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:01.318 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:01.318 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:01.318 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:01.318 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:01.319 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:01.319 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:01.319 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:01.319 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:01.319 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:01.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.317+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fc458003f40 con 0x7fc46014ff40 2024-03-29T11:39:01.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.318+0000 7fc45effd640 10 monclient: get_auth_request con 0x7fc44c06de70 auth_method 0 2024-03-29T11:39:01.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.318+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fc458084f20 con 0x7fc46014ff40 2024-03-29T11:39:01.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.318+0000 7fc4477fe640 10 client.4328.objecter ms_dispatch 0x7fc460000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:01.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.318+0000 7fc4477fe640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:01.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.318+0000 7fc4477fe640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:01.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.318+0000 7fc4477fe640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:39:01.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.318+0000 7fc4477fe640 10 client.4328.objecter ms_handle_connect 0x7fc44c06de70 2024-03-29T11:39:01.328 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.327+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc458050630 con 0x7fc46014ff40 2024-03-29T11:39:01.328 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.328+0000 7fc4477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:01.329 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.328+0000 7fc4477fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:01.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.658+0000 7fc4657bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:01.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.658+0000 7fc4657bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:01.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.658+0000 7fc4657bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:01.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.658+0000 7fc4657bc640 1 -- 172.21.15.100:0/175734511 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc440005330 con 0x7fc46014ff40 2024-03-29T11:39:01.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.659+0000 7fc4477fe640 1 -- 172.21.15.100:0/175734511 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fc458052730 con 0x7fc46014ff40 2024-03-29T11:39:01.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.659+0000 7fc4477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:01.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.659+0000 7fc4477fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:01.661 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:39:01.662 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:01.664 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.663+0000 7fc4657bc640 10 librados: watch_flush enter 2024-03-29T11:39:01.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.663+0000 7fc4657bc640 10 librados: watch_flush exit 2024-03-29T11:39:01.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.663+0000 7fc4657bc640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:39:01.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.663+0000 7fc4657bc640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:39:01.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.664+0000 7fc4657bc640 1 -- 172.21.15.100:0/175734511 >> v1:172.21.15.100:6816/3220289657 conn(0x7fc44c06de70 legacy=0x7fc44c070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:01.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.664+0000 7fc4657bc640 10 monclient: shutdown 2024-03-29T11:39:01.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.664+0000 7fc4657bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:01.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.664+0000 7fc4657bc640 1 -- 172.21.15.100:0/175734511 >> v1:172.21.15.100:6790/0 conn(0x7fc46014ff40 legacy=0x7fc460148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:01.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.664+0000 7fc4657bc640 1 -- 172.21.15.100:0/175734511 shutdown_connections 2024-03-29T11:39:01.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.664+0000 7fc4657bc640 1 -- 172.21.15.100:0/175734511 >> 172.21.15.100:0/175734511 conn(0x7fc4600b35e0 msgr2=0x7fc460140a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:01.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.664+0000 7fc4657bc640 1 -- 172.21.15.100:0/175734511 shutdown_connections 2024-03-29T11:39:01.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.665+0000 7fc4657bc640 1 -- 172.21.15.100:0/175734511 wait complete. 2024-03-29T11:39:01.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:01.665+0000 7fc4657bc640 1 librados: shutdown 2024-03-29T11:39:01.687 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:08.492 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:08.640 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.639+0000 7f333175a640 10 monclient: get_monmap_and_config 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.639+0000 7f333175a640 10 monclient: build_initial_monmap 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.639+0000 7f333175a640 10 monclient: monmap: 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:39:08.640861+0000 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:39:08.640861+0000 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:39:08.641 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:08.642 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.641+0000 7f333175a640 1 Processor -- start 2024-03-29T11:39:08.642 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.641+0000 7f333175a640 1 -- start start 2024-03-29T11:39:08.642 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.641+0000 7f333175a640 10 monclient: init 2024-03-29T11:39:08.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.642+0000 7f333175a640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:08.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.642+0000 7f333175a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:08.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.642+0000 7f333175a640 10 monclient(hunting): picked mon.noname-b con 0x7f33240389e0 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:08.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.642+0000 7f333175a640 10 monclient(hunting): picked mon.noname-a con 0x7f3324038dc0 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:08.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.642+0000 7f333175a640 10 monclient(hunting): picked mon.noname-c con 0x7f3324030660 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:08.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.642+0000 7f333175a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33240c54c0 con 0x7f3324038dc0 2024-03-29T11:39:08.644 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.642+0000 7f333175a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3324036430 con 0x7f3324030660 2024-03-29T11:39:08.644 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.642+0000 7f333175a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3324039840 con 0x7f33240389e0 2024-03-29T11:39:08.644 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.642+0000 7f333175a640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:08.644 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.642+0000 7f333175a640 10 monclient(hunting): authenticate will time out at 1068.837366s 2024-03-29T11:39:08.644 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.643+0000 7f332bfff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f33240389e0 0x7f3324033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:36964/0 (socket says 172.21.15.100:36964) 2024-03-29T11:39:08.644 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.643+0000 7f332bfff640 1 -- 172.21.15.100:0/230318357 learned_addr learned my addr 172.21.15.100:0/230318357 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:08.645 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.644+0000 7f332affd640 1 -- 172.21.15.100:0/230318357 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58091390 0 0) 0x7f3324039840 con 0x7f33240389e0 2024-03-29T11:39:08.645 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.644+0000 7f332affd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:08.645 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.644+0000 7f332affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:08.645 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.644+0000 7f332affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:08.645 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.644+0000 7f332affd640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:39:08.645 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.644+0000 7f332affd640 1 -- 172.21.15.100:0/230318357 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f331c003520 con 0x7f33240389e0 2024-03-29T11:39:08.645 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.644+0000 7f332affd640 1 -- 172.21.15.100:0/230318357 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 357551713 0 0) 0x7f331c003520 con 0x7f33240389e0 2024-03-29T11:39:08.645 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.644+0000 7f332affd640 1 -- 172.21.15.100:0/230318357 >> v1:172.21.15.191:6789/0 conn(0x7f3324030660 legacy=0x7f33240d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:08.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.645+0000 7f332affd640 1 -- 172.21.15.100:0/230318357 >> v1:172.21.15.100:6789/0 conn(0x7f3324038dc0 legacy=0x7f33240cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:08.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.645+0000 7f332affd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:08.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.645+0000 7f332affd640 1 monclient: found mon.noname-b 2024-03-29T11:39:08.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.645+0000 7f332affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:08.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.645+0000 7f332affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:39:08.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.645+0000 7f332affd640 1 -- 172.21.15.100:0/230318357 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3324035690 con 0x7f33240389e0 2024-03-29T11:39:08.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.645+0000 7f332affd640 10 monclient: _finish_auth 0 2024-03-29T11:39:08.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.645+0000 7f332affd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:08.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.645+0000 7f332affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f332affd640 1 -- 172.21.15.100:0/230318357 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3320001690 con 0x7f33240389e0 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f332affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f332affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f332affd640 10 monclient: dump: 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f332affd640 1 -- 172.21.15.100:0/230318357 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3320001920 con 0x7f33240389e0 2024-03-29T11:39:08.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f332affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f332affd640 1 -- 172.21.15.100:0/230318357 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3320001cf0 con 0x7f33240389e0 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f332affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f332affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f332affd640 10 monclient: dump: 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:08.650 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:08.651 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f333175a640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:39:08.651 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f333175a640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:08.651 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.648+0000 7f333175a640 4 set_mon_vals no callback set 2024-03-29T11:39:08.651 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.649+0000 7f333175a640 10 monclient: shutdown 2024-03-29T11:39:08.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.649+0000 7f333175a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:08.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.649+0000 7f333175a640 1 -- 172.21.15.100:0/230318357 >> v1:172.21.15.100:6790/0 conn(0x7f33240389e0 legacy=0x7f3324033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:08.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.649+0000 7f333175a640 1 -- 172.21.15.100:0/230318357 shutdown_connections 2024-03-29T11:39:08.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.649+0000 7f333175a640 1 -- 172.21.15.100:0/230318357 >> 172.21.15.100:0/230318357 conn(0x7f3324005c00 msgr2=0x7f3324006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:08.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.651+0000 7f333175a640 1 -- 172.21.15.100:0/230318357 shutdown_connections 2024-03-29T11:39:08.653 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.652+0000 7f333175a640 1 -- 172.21.15.100:0/230318357 wait complete. 2024-03-29T11:39:08.653 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.653+0000 7f333175a640 10 monclient: build_initial_monmap 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.653+0000 7f333175a640 10 monclient: monmap: 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:39:08.654 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:08.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.653+0000 7f333175a640 1 librados: starting msgr at 2024-03-29T11:39:08.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.653+0000 7f333175a640 1 librados: starting objecter 2024-03-29T11:39:08.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.653+0000 7f333175a640 1 Processor -- start 2024-03-29T11:39:08.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.654+0000 7f333175a640 1 -- start start 2024-03-29T11:39:08.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.654+0000 7f333175a640 1 librados: setting wanted keys 2024-03-29T11:39:08.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.654+0000 7f333175a640 1 librados: calling monclient init 2024-03-29T11:39:08.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.654+0000 7f333175a640 10 monclient: init 2024-03-29T11:39:08.656 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.654+0000 7f333175a640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:08.656 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.654+0000 7f333175a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:08.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.657+0000 7f333175a640 10 monclient(hunting): picked mon.noname-b con 0x7f3324030660 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:08.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.657+0000 7f333175a640 10 monclient(hunting): picked mon.noname-c con 0x7f33240389e0 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:08.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.657+0000 7f333175a640 10 monclient(hunting): picked mon.noname-a con 0x7f33240c7530 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:08.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.657+0000 7f333175a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3324160c50 con 0x7f33240c7530 2024-03-29T11:39:08.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.657+0000 7f333175a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33240c2cc0 con 0x7f3324030660 2024-03-29T11:39:08.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.657+0000 7f333175a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33241608e0 con 0x7f33240389e0 2024-03-29T11:39:08.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.657+0000 7f333175a640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:08.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.657+0000 7f333175a640 10 monclient(hunting): authenticate will time out at 1068.852953s 2024-03-29T11:39:08.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.658+0000 7f332bfff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3324030660 0x7f33241601d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:37533/0 (socket says 172.21.15.100:37533) 2024-03-29T11:39:08.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.658+0000 7f332bfff640 1 -- 172.21.15.100:0/1701585446 learned_addr learned my addr 172.21.15.100:0/1701585446 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:08.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.658+0000 7f3328ff9640 10 client.?.objecter ms_handle_connect 0x7f3324030660 2024-03-29T11:39:08.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.658+0000 7f3328ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:08.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.658+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002614958 0 0) 0x7f33240c2cc0 con 0x7f3324030660 2024-03-29T11:39:08.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:39:08.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:08.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:08.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:39:08.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3310004eb0 con 0x7f3324030660 2024-03-29T11:39:08.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3457027323 0 0) 0x7f3310004eb0 con 0x7f3324030660 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33240c2cc0 con 0x7f3324030660 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3320001610 con 0x7f3324030660 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 10 monclient(hunting): dump: 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:08.661 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:08.662 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:08.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.659+0000 7f3328ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:08.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2502406561 0 0) 0x7f33240c2cc0 con 0x7f3324030660 2024-03-29T11:39:08.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 >> v1:172.21.15.100:6790/0 conn(0x7f33240389e0 legacy=0x7f33240c1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:08.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 >> v1:172.21.15.100:6789/0 conn(0x7f33240c7530 legacy=0x7f332402d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:08.664 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:39:08.664 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 1 monclient: found mon.b 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33240c2670 con 0x7f3324030660 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 10 monclient: _finish_auth 0 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.660+0000 7f3328ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3320004ca0 con 0x7f3324030660 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f3328ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f33200050b0 con 0x7f3324030660 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f3328ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f3328ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f3328ff9640 10 monclient: dump: 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:08.665 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f333175a640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f333175a640 10 monclient: _renew_subs 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f333175a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f333175a640 1 -- 172.21.15.100:0/1701585446 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33240c28c0 con 0x7f3324030660 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f333175a640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f333175a640 10 monclient: _renew_subs 2024-03-29T11:39:08.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f333175a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:39:08.667 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f333175a640 1 -- 172.21.15.100:0/1701585446 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f332402dd70 con 0x7f3324030660 2024-03-29T11:39:08.667 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.661+0000 7f333175a640 1 librados: init done 2024-03-29T11:39:08.667 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.662+0000 7f332a7fc640 4 set_mon_vals no callback set 2024-03-29T11:39:08.668 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.664+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f3320005210 con 0x7f3324030660 2024-03-29T11:39:08.668 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.665+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f3320084d00 con 0x7f3324030660 2024-03-29T11:39:08.668 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.665+0000 7f3328ff9640 10 client.4324.objecter ms_dispatch 0x7f332c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:08.668 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.665+0000 7f3328ff9640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:08.668 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.665+0000 7f3328ff9640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:08.668 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.665+0000 7f3328ff9640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-29T11:39:08.668 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.665+0000 7f332b7fe640 10 monclient: get_auth_request con 0x7f331006dce0 auth_method 0 2024-03-29T11:39:08.668 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.666+0000 7f3328ff9640 10 client.4324.objecter ms_handle_connect 0x7f331006dce0 2024-03-29T11:39:08.668 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.667+0000 7f333175a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:08.669 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.667+0000 7f333175a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:08.669 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.667+0000 7f333175a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:39:08.669 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.667+0000 7f333175a640 1 -- 172.21.15.100:0/1701585446 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330c005080 con 0x7f3324030660 2024-03-29T11:39:08.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.675+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33200505e0 con 0x7f3324030660 2024-03-29T11:39:08.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.675+0000 7f3328ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:08.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.676+0000 7f3328ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:08.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.995+0000 7f333175a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:08.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.995+0000 7f333175a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:08.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.995+0000 7f333175a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:39:08.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.995+0000 7f333175a640 1 -- 172.21.15.100:0/1701585446 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f330c005330 con 0x7f3324030660 2024-03-29T11:39:08.997 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.996+0000 7f3328ff9640 1 -- 172.21.15.100:0/1701585446 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f3320052510 con 0x7f3324030660 2024-03-29T11:39:08.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.996+0000 7f3328ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:08.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:08.996+0000 7f3328ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:09.000 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:39:09.000 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:09.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.002+0000 7f333175a640 10 librados: watch_flush enter 2024-03-29T11:39:09.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.004+0000 7f333175a640 10 librados: watch_flush exit 2024-03-29T11:39:09.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.004+0000 7f333175a640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-29T11:39:09.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.004+0000 7f333175a640 10 client.4324.objecter successfully canceled tick 2024-03-29T11:39:09.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.004+0000 7f333175a640 1 -- 172.21.15.100:0/1701585446 >> v1:172.21.15.100:6816/3220289657 conn(0x7f331006dce0 legacy=0x7f33100701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:09.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.004+0000 7f333175a640 10 monclient: shutdown 2024-03-29T11:39:09.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.004+0000 7f333175a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:09.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.004+0000 7f333175a640 1 -- 172.21.15.100:0/1701585446 >> v1:172.21.15.191:6789/0 conn(0x7f3324030660 legacy=0x7f33241601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:09.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.005+0000 7f333175a640 1 -- 172.21.15.100:0/1701585446 shutdown_connections 2024-03-29T11:39:09.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.005+0000 7f333175a640 1 -- 172.21.15.100:0/1701585446 >> 172.21.15.100:0/1701585446 conn(0x7f3324005c00 msgr2=0x7f33240c36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:09.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.005+0000 7f333175a640 1 -- 172.21.15.100:0/1701585446 shutdown_connections 2024-03-29T11:39:09.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.005+0000 7f333175a640 1 -- 172.21.15.100:0/1701585446 wait complete. 2024-03-29T11:39:09.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:09.005+0000 7f333175a640 1 librados: shutdown 2024-03-29T11:39:09.023 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:15.830 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:15.960 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.959+0000 7fedb273a640 10 monclient: get_monmap_and_config 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.959+0000 7fedb273a640 10 monclient: build_initial_monmap 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.959+0000 7fedb273a640 10 monclient: monmap: 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:39:15.960754+0000 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:39:15.960754+0000 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:39:15.961 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:15.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.961+0000 7fedb273a640 1 Processor -- start 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.961+0000 7fedb273a640 1 -- start start 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.961+0000 7fedb273a640 10 monclient: init 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb273a640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb273a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb273a640 10 monclient(hunting): picked mon.noname-a con 0x7fedac1457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb273a640 10 monclient(hunting): picked mon.noname-c con 0x7fedac13f650 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb273a640 10 monclient(hunting): picked mon.noname-b con 0x7fedac0a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb273a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac0aaef0 con 0x7fedac1457f0 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb273a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac145bd0 con 0x7fedac13f650 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb273a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac145eb0 con 0x7fedac0a9d70 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb273a640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:15.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb273a640 10 monclient(hunting): authenticate will time out at 1076.157258s 2024-03-29T11:39:15.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb0cd7640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fedac0a9d70 0x7fedac14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:34702/0 (socket says 172.21.15.100:34702) 2024-03-29T11:39:15.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedb0cd7640 1 -- 172.21.15.100:0/3013785695 learned_addr learned my addr 172.21.15.100:0/3013785695 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:15.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7feda37fe640 1 --1- 172.21.15.100:0/3013785695 >> v1:172.21.15.191:6789/0 conn(0x7fedac13f650 0x7fedac0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:38084/0 (socket says 172.21.15.100:38084) 2024-03-29T11:39:15.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedab7fe640 1 -- 172.21.15.100:0/3013785695 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199735660 0 0) 0x7fedac145eb0 con 0x7fedac0a9d70 2024-03-29T11:39:15.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:15.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:15.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:15.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedab7fe640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:39:15.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedab7fe640 1 -- 172.21.15.100:0/3013785695 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed98003520 con 0x7fedac0a9d70 2024-03-29T11:39:15.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedab7fe640 1 -- 172.21.15.100:0/3013785695 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802241950 0 0) 0x7fed98003520 con 0x7fedac0a9d70 2024-03-29T11:39:15.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedab7fe640 1 -- 172.21.15.100:0/3013785695 >> v1:172.21.15.191:6789/0 conn(0x7fedac13f650 legacy=0x7fedac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:15.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.962+0000 7fedab7fe640 1 -- 172.21.15.100:0/3013785695 >> v1:172.21.15.100:6789/0 conn(0x7fedac1457f0 legacy=0x7fedac13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:15.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 1 monclient: found mon.noname-b 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 1 -- 172.21.15.100:0/3013785695 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac146340 con 0x7fedac0a9d70 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedb273a640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedb273a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 1 -- 172.21.15.100:0/3013785695 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7feda4001a10 con 0x7fedac0a9d70 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:15.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedab7fe640 10 monclient: dump: 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:15.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.965+0000 7fedb273a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:15.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.966+0000 7fedab7fe640 1 -- 172.21.15.100:0/3013785695 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feda4001ce0 con 0x7fedac0a9d70 2024-03-29T11:39:15.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.966+0000 7fedab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:15.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.966+0000 7fedb273a640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:15.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.966+0000 7fedb273a640 4 set_mon_vals no callback set 2024-03-29T11:39:15.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedab7fe640 1 -- 172.21.15.100:0/3013785695 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7feda4002070 con 0x7fedac0a9d70 2024-03-29T11:39:15.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:15.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:15.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedab7fe640 10 monclient: dump: 2024-03-29T11:39:15.970 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:15.970 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedb273a640 10 monclient: shutdown 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedb273a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedb273a640 1 -- 172.21.15.100:0/3013785695 >> v1:172.21.15.100:6790/0 conn(0x7fedac0a9d70 legacy=0x7fedac14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedb273a640 1 -- 172.21.15.100:0/3013785695 shutdown_connections 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedb273a640 1 -- 172.21.15.100:0/3013785695 >> 172.21.15.100:0/3013785695 conn(0x7fedac066a00 msgr2=0x7fedac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:15.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedb273a640 1 -- 172.21.15.100:0/3013785695 shutdown_connections 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.967+0000 7fedb273a640 1 -- 172.21.15.100:0/3013785695 wait complete. 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.968+0000 7fedb273a640 10 monclient: build_initial_monmap 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.968+0000 7fedb273a640 10 monclient: monmap: 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:15.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.969+0000 7fedb273a640 1 librados: starting msgr at 2024-03-29T11:39:15.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.969+0000 7fedb273a640 1 librados: starting objecter 2024-03-29T11:39:15.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.969+0000 7fedb273a640 1 Processor -- start 2024-03-29T11:39:15.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.969+0000 7fedb273a640 1 -- start start 2024-03-29T11:39:15.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.969+0000 7fedb273a640 1 librados: setting wanted keys 2024-03-29T11:39:15.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.969+0000 7fedb273a640 1 librados: calling monclient init 2024-03-29T11:39:15.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.969+0000 7fedb273a640 10 monclient: init 2024-03-29T11:39:15.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.970+0000 7fedb273a640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:15.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.970+0000 7fedb273a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:15.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.970+0000 7fedb273a640 10 monclient(hunting): picked mon.noname-b con 0x7fedac0a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:15.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.970+0000 7fedb273a640 10 monclient(hunting): picked mon.noname-a con 0x7fedac13f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:15.979 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.970+0000 7fedabfff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fedac0a9d70 0x7fedac1db6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:39372/0 (socket says 172.21.15.100:39372) 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.970+0000 7fedabfff640 1 -- 172.21.15.100:0/1783358842 learned_addr learned my addr 172.21.15.100:0/1783358842 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7fedb273a640 10 monclient(hunting): picked mon.noname-c con 0x7fedac1457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac141f70 con 0x7fedac13f650 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac06ed90 con 0x7fedac0a9d70 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7feda97fa640 10 client.?.objecter ms_handle_connect 0x7fedac0a9d70 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7feda97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedac1447b0 con 0x7fedac1457f0 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7fedb273a640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7fedb273a640 10 monclient(hunting): authenticate will time out at 1076.166584s 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208275009 0 0) 0x7fedac06ed90 con 0x7fedac0a9d70 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7feda97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7feda97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:15.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7feda97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7feda97fa640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.971+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed8c004eb0 con 0x7fedac0a9d70 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 10 client.?.objecter ms_handle_connect 0x7fedac1457f0 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308484934 0 0) 0x7fedac1447b0 con 0x7fedac1457f0 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedac06ed90 con 0x7fedac1457f0 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1188941427 0 0) 0x7fedac06ed90 con 0x7fedac1457f0 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedac1447b0 con 0x7fedac1457f0 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7feda40019d0 con 0x7fedac1457f0 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:15.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 10 monclient(hunting): dump: 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:15.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 564609283 0 0) 0x7fedac1447b0 con 0x7fedac1457f0 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 >> v1:172.21.15.191:6789/0 conn(0x7fedac0a9d70 legacy=0x7fedac1db6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.972+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 >> v1:172.21.15.100:6789/0 conn(0x7fedac13f650 legacy=0x7fedac1426e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 1 monclient: found mon.c 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac144940 con 0x7fedac1457f0 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7fedb273a640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7fedb273a640 10 monclient: _renew_subs 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7fedb273a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac071670 con 0x7fedac1457f0 2024-03-29T11:39:15.983 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feda4001360 con 0x7fedac1457f0 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7feda4005150 con 0x7fedac1457f0 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7feda97fa640 10 monclient: dump: 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:15.984 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.974+0000 7fedaaffd640 4 set_mon_vals no callback set 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.976+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7feda401b120 con 0x7fedac1457f0 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.976+0000 7fedb273a640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.976+0000 7fedb273a640 10 monclient: _renew_subs 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.976+0000 7fedb273a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.976+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac071bf0 con 0x7fedac1457f0 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.976+0000 7fedb273a640 1 librados: init done 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.977+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7feda4085030 con 0x7fedac1457f0 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.977+0000 7feda97fa640 10 client.4337.objecter ms_dispatch 0x7fedac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.977+0000 7feda97fa640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.977+0000 7feda97fa640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.977+0000 7feda97fa640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:39:15.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.978+0000 7fedabfff640 10 monclient: get_auth_request con 0x7fed8c06dce0 auth_method 0 2024-03-29T11:39:15.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.979+0000 7feda97fa640 10 client.4337.objecter ms_handle_connect 0x7fed8c06dce0 2024-03-29T11:39:15.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.980+0000 7fedb273a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:15.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.980+0000 7fedb273a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:15.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.980+0000 7fedb273a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:15.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.980+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedac06ebd0 con 0x7fedac1457f0 2024-03-29T11:39:15.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.987+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feda40507a0 con 0x7fedac1457f0 2024-03-29T11:39:15.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.987+0000 7feda97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:15.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:15.987+0000 7feda97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:16.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.298+0000 7fedb273a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:16.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.298+0000 7fedb273a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:16.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.298+0000 7fedb273a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:16.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.298+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedac071800 con 0x7fedac1457f0 2024-03-29T11:39:16.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.299+0000 7feda97fa640 1 -- 172.21.15.100:0/1783358842 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7feda40846e0 con 0x7fedac1457f0 2024-03-29T11:39:16.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.299+0000 7feda97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:16.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.299+0000 7feda97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:16.301 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:39:16.301 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 10 librados: watch_flush enter 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 10 librados: watch_flush exit 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 >> v1:172.21.15.100:6816/3220289657 conn(0x7fed8c06dce0 legacy=0x7fed8c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 10 monclient: shutdown 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 >> v1:172.21.15.100:6790/0 conn(0x7fedac1457f0 legacy=0x7fedac141860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 shutdown_connections 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 >> 172.21.15.100:0/1783358842 conn(0x7fedac066a00 msgr2=0x7fedac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 shutdown_connections 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 1 -- 172.21.15.100:0/1783358842 wait complete. 2024-03-29T11:39:16.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:16.303+0000 7fedb273a640 1 librados: shutdown 2024-03-29T11:39:16.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:23.126 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.272+0000 7f86782f1640 10 monclient: get_monmap_and_config 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.272+0000 7f86782f1640 10 monclient: build_initial_monmap 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.272+0000 7f86782f1640 10 monclient: monmap: 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:39:23.273746+0000 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:39:23.273746+0000 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:23.274 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:39:23.275 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:39:23.275 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:23.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.273+0000 7f86782f1640 1 Processor -- start 2024-03-29T11:39:23.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.273+0000 7f86782f1640 1 -- start start 2024-03-29T11:39:23.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.274+0000 7f86782f1640 10 monclient: init 2024-03-29T11:39:23.278 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.274+0000 7f86782f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f86782f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f86782f1640 10 monclient(hunting): picked mon.noname-a con 0x7f86701457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f86782f1640 10 monclient(hunting): picked mon.noname-b con 0x7f867013f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f86782f1640 10 monclient(hunting): picked mon.noname-c con 0x7f86700a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f86782f1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86700aaef0 con 0x7f86701457f0 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f86782f1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8670145bd0 con 0x7f86700a9d70 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f86782f1640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8670145eb0 con 0x7f867013f650 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f86782f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f86782f1640 10 monclient(hunting): authenticate will time out at 1083.470163s 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f867588c640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f867013f650 0x7f86700b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:54126/0 (socket says 172.21.15.100:54126) 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.275+0000 7f867588c640 1 -- 172.21.15.100:0/3569274028 learned_addr learned my addr 172.21.15.100:0/3569274028 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.276+0000 7f867508b640 1 -- 172.21.15.100:0/3569274028 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942580878 0 0) 0x7f8670145eb0 con 0x7f867013f650 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.276+0000 7f867508b640 10 monclient(hunting): _negotiate 2024-03-29T11:39:23.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.276+0000 7f867508b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:23.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.276+0000 7f867508b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:23.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.276+0000 7f867508b640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:39:23.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.277+0000 7f867508b640 1 -- 172.21.15.100:0/3569274028 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f865c003520 con 0x7f867013f650 2024-03-29T11:39:23.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.277+0000 7f867508b640 1 -- 172.21.15.100:0/3569274028 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 548683543 0 0) 0x7f865c003520 con 0x7f867013f650 2024-03-29T11:39:23.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.277+0000 7f867508b640 1 -- 172.21.15.100:0/3569274028 >> v1:172.21.15.191:6789/0 conn(0x7f86700a9d70 legacy=0x7f867014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:23.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.279+0000 7f867508b640 1 -- 172.21.15.100:0/3569274028 >> v1:172.21.15.100:6789/0 conn(0x7f86701457f0 legacy=0x7f867013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:23.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.279+0000 7f867508b640 10 monclient: _finish_hunting 0 2024-03-29T11:39:23.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.279+0000 7f867508b640 1 monclient: found mon.noname-b 2024-03-29T11:39:23.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.279+0000 7f867508b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:23.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.279+0000 7f867508b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:39:23.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f867508b640 1 -- 172.21.15.100:0/3569274028 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8670146340 con 0x7f867013f650 2024-03-29T11:39:23.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f867508b640 10 monclient: _finish_auth 0 2024-03-29T11:39:23.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f867508b640 10 monclient: _check_auth_tickets 2024-03-29T11:39:23.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f867508b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f867508b640 1 -- 172.21.15.100:0/3569274028 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8668001a10 con 0x7f867013f650 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f867508b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f867508b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f867508b640 10 monclient: dump: 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:23.283 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f86782f1640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f86782f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f867508b640 1 -- 172.21.15.100:0/3569274028 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8668001ca0 con 0x7f867013f650 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f867508b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f86782f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.280+0000 7f86782f1640 4 set_mon_vals no callback set 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.281+0000 7f867508b640 1 -- 172.21.15.100:0/3569274028 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8668002030 con 0x7f867013f650 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.281+0000 7f867508b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.281+0000 7f867508b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.281+0000 7f867508b640 10 monclient: dump: 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:23.284 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:23.285 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:23.285 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:23.285 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:23.285 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:23.285 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:23.285 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:23.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.281+0000 7f86782f1640 10 monclient: shutdown 2024-03-29T11:39:23.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.281+0000 7f86782f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:23.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.281+0000 7f86782f1640 1 -- 172.21.15.100:0/3569274028 >> v1:172.21.15.100:6790/0 conn(0x7f867013f650 legacy=0x7f86700b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:23.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.282+0000 7f86782f1640 1 -- 172.21.15.100:0/3569274028 shutdown_connections 2024-03-29T11:39:23.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.282+0000 7f86782f1640 1 -- 172.21.15.100:0/3569274028 >> 172.21.15.100:0/3569274028 conn(0x7f8670066a00 msgr2=0x7f8670066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:23.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.282+0000 7f86782f1640 1 -- 172.21.15.100:0/3569274028 shutdown_connections 2024-03-29T11:39:23.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.282+0000 7f86782f1640 1 -- 172.21.15.100:0/3569274028 wait complete. 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.283+0000 7f86782f1640 10 monclient: build_initial_monmap 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.283+0000 7f86782f1640 10 monclient: monmap: 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.283+0000 7f86782f1640 1 librados: starting msgr at 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.283+0000 7f86782f1640 1 librados: starting objecter 2024-03-29T11:39:23.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.283+0000 7f86782f1640 1 Processor -- start 2024-03-29T11:39:23.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.283+0000 7f86782f1640 1 -- start start 2024-03-29T11:39:23.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.283+0000 7f86782f1640 1 librados: setting wanted keys 2024-03-29T11:39:23.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 1 librados: calling monclient init 2024-03-29T11:39:23.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 10 monclient: init 2024-03-29T11:39:23.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:23.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:23.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 10 monclient(hunting): picked mon.noname-a con 0x7f86700a9d70 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:23.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 10 monclient(hunting): picked mon.noname-c con 0x7f867013f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:23.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 10 monclient(hunting): picked mon.noname-b con 0x7f86701457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:23.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8670142b10 con 0x7f86700a9d70 2024-03-29T11:39:23.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8670142da0 con 0x7f86701457f0 2024-03-29T11:39:23.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86701420a0 con 0x7f867013f650 2024-03-29T11:39:23.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.284+0000 7f86782f1640 10 monclient(hunting): authenticate will time out at 1083.479925s 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.285+0000 7f867588c640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f867013f650 0x7f86700b0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:60332/0 (socket says 172.21.15.100:60332) 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.285+0000 7f867588c640 1 -- 172.21.15.100:0/1851217201 learned_addr learned my addr 172.21.15.100:0/1851217201 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.287+0000 7f8666ffd640 10 client.?.objecter ms_handle_connect 0x7f867013f650 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.287+0000 7f8666ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.287+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967546837 0 0) 0x7f86701420a0 con 0x7f867013f650 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.288+0000 7f8666ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.288+0000 7f8666ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.288+0000 7f8666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.288+0000 7f8666ffd640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:39:23.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.288+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8654004eb0 con 0x7f867013f650 2024-03-29T11:39:23.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.289+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 248570353 0 0) 0x7f8654004eb0 con 0x7f867013f650 2024-03-29T11:39:23.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.289+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86701420a0 con 0x7f867013f650 2024-03-29T11:39:23.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.289+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8668001990 con 0x7f867013f650 2024-03-29T11:39:23.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.289+0000 7f8666ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:23.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.289+0000 7f8666ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.289+0000 7f8666ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.289+0000 7f8666ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.289+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1883041419 0 0) 0x7f86701420a0 con 0x7f867013f650 2024-03-29T11:39:23.294 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.289+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 >> v1:172.21.15.191:6789/0 conn(0x7f86701457f0 legacy=0x7f86701da810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.290+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 >> v1:172.21.15.100:6789/0 conn(0x7f86700a9d70 legacy=0x7f8670142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.290+0000 7f8666ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.290+0000 7f8666ffd640 1 monclient: found mon.c 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.290+0000 7f8666ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f8666ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8670144800 con 0x7f867013f650 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f8666ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f8666ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f8666ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f86782f1640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f86782f1640 10 monclient: _renew_subs 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f86782f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f86782f1640 1 -- 172.21.15.100:0/1851217201 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8670144a50 con 0x7f867013f650 2024-03-29T11:39:23.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f86782f1640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:23.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f86782f1640 10 monclient: _renew_subs 2024-03-29T11:39:23.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f86782f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:23.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f86782f1640 1 -- 172.21.15.100:0/1851217201 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86700b16a0 con 0x7f867013f650 2024-03-29T11:39:23.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.291+0000 7f86782f1640 1 librados: init done 2024-03-29T11:39:23.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.293+0000 7f86782f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.293+0000 7f86782f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.293+0000 7f86782f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.293+0000 7f86782f1640 1 -- 172.21.15.100:0/1851217201 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86701da530 con 0x7f867013f650 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.294+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8668004fe0 con 0x7f867013f650 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.294+0000 7f8666ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.294+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8668005250 con 0x7f867013f650 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.294+0000 7f8666ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.294+0000 7f8666ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.294+0000 7f8666ffd640 10 monclient: dump: 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:23.297 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:23.298 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:23.298 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:23.298 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:23.298 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:23.298 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:23.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.295+0000 7f867488a640 4 set_mon_vals no callback set 2024-03-29T11:39:23.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.298+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f8668005470 con 0x7f867013f650 2024-03-29T11:39:23.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.299+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f8668084e70 con 0x7f867013f650 2024-03-29T11:39:23.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.299+0000 7f8666ffd640 10 client.4343.objecter ms_dispatch 0x7f8670000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:23.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.299+0000 7f8666ffd640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:23.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.299+0000 7f8666ffd640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:23.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.299+0000 7f8666ffd640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T11:39:23.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.299+0000 7f867608d640 10 monclient: get_auth_request con 0x7f865406dc40 auth_method 0 2024-03-29T11:39:23.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.300+0000 7f8666ffd640 10 client.4343.objecter ms_handle_connect 0x7f865406dc40 2024-03-29T11:39:23.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.304+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8668050750 con 0x7f867013f650 2024-03-29T11:39:23.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.304+0000 7f8666ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:23.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.304+0000 7f8666ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:23.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.632+0000 7f86782f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:23.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.632+0000 7f86782f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:23.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.632+0000 7f86782f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:23.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.632+0000 7f86782f1640 1 -- 172.21.15.100:0/1851217201 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8670075f10 con 0x7f867013f650 2024-03-29T11:39:23.635 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.634+0000 7f8666ffd640 1 -- 172.21.15.100:0/1851217201 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f8668052680 con 0x7f867013f650 2024-03-29T11:39:23.635 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.634+0000 7f8666ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:23.635 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.634+0000 7f8666ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:23.637 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:39:23.637 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:23.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.638+0000 7f86782f1640 10 librados: watch_flush enter 2024-03-29T11:39:23.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.638+0000 7f86782f1640 10 librados: watch_flush exit 2024-03-29T11:39:23.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.638+0000 7f86782f1640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T11:39:23.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.638+0000 7f86782f1640 10 client.4343.objecter successfully canceled tick 2024-03-29T11:39:23.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.638+0000 7f86782f1640 1 -- 172.21.15.100:0/1851217201 >> v1:172.21.15.100:6816/3220289657 conn(0x7f865406dc40 legacy=0x7f8654070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:23.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.638+0000 7f86782f1640 10 monclient: shutdown 2024-03-29T11:39:23.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.638+0000 7f86782f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:23.640 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.638+0000 7f86782f1640 1 -- 172.21.15.100:0/1851217201 >> v1:172.21.15.100:6790/0 conn(0x7f867013f650 legacy=0x7f86700b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:23.640 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.638+0000 7f86782f1640 1 -- 172.21.15.100:0/1851217201 shutdown_connections 2024-03-29T11:39:23.640 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.638+0000 7f86782f1640 1 -- 172.21.15.100:0/1851217201 >> 172.21.15.100:0/1851217201 conn(0x7f8670066a00 msgr2=0x7f86700b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:23.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.642+0000 7f86782f1640 1 -- 172.21.15.100:0/1851217201 shutdown_connections 2024-03-29T11:39:23.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.642+0000 7f86782f1640 1 -- 172.21.15.100:0/1851217201 wait complete. 2024-03-29T11:39:23.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:23.642+0000 7f86782f1640 1 librados: shutdown 2024-03-29T11:39:23.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:30.467 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:30.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.623+0000 7f6ff76cf640 10 monclient: get_monmap_and_config 2024-03-29T11:39:30.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.623+0000 7f6ff76cf640 10 monclient: build_initial_monmap 2024-03-29T11:39:30.624 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.623+0000 7f6ff76cf640 10 monclient: monmap: 2024-03-29T11:39:30.624 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:30.624 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:30.625 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:39:30.624104+0000 2024-03-29T11:39:30.625 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:39:30.624104+0000 2024-03-29T11:39:30.625 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:30.625 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:30.625 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:30.625 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:39:30.625 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:39:30.625 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:30.625 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.624+0000 7f6ff76cf640 1 Processor -- start 2024-03-29T11:39:30.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.624+0000 7f6ff76cf640 1 -- start start 2024-03-29T11:39:30.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.624+0000 7f6ff76cf640 10 monclient: init 2024-03-29T11:39:30.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.625+0000 7f6ff76cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:30.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.625+0000 7f6ff76cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:30.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.627+0000 7f6ff76cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff00b0860 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:30.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.627+0000 7f6ff76cf640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff013ef30 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:30.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.627+0000 7f6ff76cf640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff00af8e0 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:30.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.627+0000 7f6ff76cf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff00b0c40 con 0x7f6ff00af8e0 2024-03-29T11:39:30.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.627+0000 7f6ff76cf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff00aed10 con 0x7f6ff013ef30 2024-03-29T11:39:30.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.627+0000 7f6ff76cf640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff00afd50 con 0x7f6ff00b0860 2024-03-29T11:39:30.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.627+0000 7f6ff76cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:30.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.627+0000 7f6ff76cf640 10 monclient(hunting): authenticate will time out at 1090.822962s 2024-03-29T11:39:30.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.631+0000 7f6ff5c6c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f6ff00af8e0 0x7f6ff0149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:59465/0 (socket says 172.21.15.100:59465) 2024-03-29T11:39:30.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.632+0000 7f6ff5c6c640 1 -- 172.21.15.100:0/1105457162 learned_addr learned my addr 172.21.15.100:0/1105457162 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.633+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478658748 0 0) 0x7f6ff00b0c40 con 0x7f6ff00af8e0 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.633+0000 7f6fe7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.633+0000 7f6fe7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.633+0000 7f6fe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.633+0000 7f6fe7fff640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.634+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fdc003520 con 0x7f6ff00af8e0 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.634+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975064590 0 0) 0x7f6ff00afd50 con 0x7f6ff00b0860 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.634+0000 7f6fe7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.634+0000 7f6fe7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.634+0000 7f6fe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:30.636 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.634+0000 7f6fe7fff640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:39:30.637 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.634+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff00b0c40 con 0x7f6ff00b0860 2024-03-29T11:39:30.638 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.635+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016416334 0 0) 0x7f6ff00aed10 con 0x7f6ff013ef30 2024-03-29T11:39:30.638 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.635+0000 7f6fe7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:30.638 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.635+0000 7f6fe7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:30.638 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.635+0000 7f6fe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:30.638 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.635+0000 7f6fe7fff640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:39:30.638 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.635+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff00afd50 con 0x7f6ff013ef30 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25417779 0 0) 0x7f6fdc003520 con 0x7f6ff00af8e0 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 >> v1:172.21.15.100:6790/0 conn(0x7f6ff00b0860 legacy=0x7f6ff013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 >> v1:172.21.15.191:6789/0 conn(0x7f6ff013ef30 legacy=0x7f6ff00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 1 monclient: found mon.noname-a 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff00ab1b0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6fe0001690 con 0x7f6ff00af8e0 2024-03-29T11:39:30.639 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6fe7fff640 10 monclient: dump: 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:30.640 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6ff76cf640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.636+0000 7f6ff76cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.637+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fe00017f0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.637+0000 7f6fe7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.637+0000 7f6fe7fff640 1 -- 172.21.15.100:0/1105457162 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6fe0001bc0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.637+0000 7f6fe7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.637+0000 7f6fe7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.637+0000 7f6fe7fff640 10 monclient: dump: 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:30.641 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:30.642 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:30.642 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:30.642 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:30.642 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:30.642 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:30.642 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.637+0000 7f6ff76cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:30.642 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.637+0000 7f6ff76cf640 4 set_mon_vals no callback set 2024-03-29T11:39:30.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.638+0000 7f6ff76cf640 10 monclient: shutdown 2024-03-29T11:39:30.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.638+0000 7f6ff76cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:30.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.638+0000 7f6ff76cf640 1 -- 172.21.15.100:0/1105457162 >> v1:172.21.15.100:6789/0 conn(0x7f6ff00af8e0 legacy=0x7f6ff0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:30.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.639+0000 7f6ff76cf640 1 -- 172.21.15.100:0/1105457162 shutdown_connections 2024-03-29T11:39:30.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.640+0000 7f6ff76cf640 1 -- 172.21.15.100:0/1105457162 >> 172.21.15.100:0/1105457162 conn(0x7f6ff006b660 msgr2=0x7f6ff006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:30.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.641+0000 7f6ff76cf640 1 -- 172.21.15.100:0/1105457162 shutdown_connections 2024-03-29T11:39:30.643 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.641+0000 7f6ff76cf640 1 -- 172.21.15.100:0/1105457162 wait complete. 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.643+0000 7f6ff76cf640 10 monclient: build_initial_monmap 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.643+0000 7f6ff76cf640 10 monclient: monmap: 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:39:30.644 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.643+0000 7f6ff76cf640 1 librados: starting msgr at 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.643+0000 7f6ff76cf640 1 librados: starting objecter 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.643+0000 7f6ff76cf640 1 Processor -- start 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.643+0000 7f6ff76cf640 1 -- start start 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.643+0000 7f6ff76cf640 1 librados: setting wanted keys 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.643+0000 7f6ff76cf640 1 librados: calling monclient init 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.643+0000 7f6ff76cf640 10 monclient: init 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.644+0000 7f6ff76cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.644+0000 7f6ff76cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.644+0000 7f6ff76cf640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff00af8e0 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.644+0000 7f6ff76cf640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff00b0860 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:30.646 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.644+0000 7f6ff76cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff013ef30 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:30.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.644+0000 7f6ff76cf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff006eb00 con 0x7f6ff00b0860 2024-03-29T11:39:30.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.644+0000 7f6ff76cf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff0070d10 con 0x7f6ff013ef30 2024-03-29T11:39:30.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.644+0000 7f6ff76cf640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff006e6b0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.644+0000 7f6ff76cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:30.647 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.644+0000 7f6ff76cf640 10 monclient(hunting): authenticate will time out at 1090.840004s 2024-03-29T11:39:30.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.648+0000 7f6ff546b640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f6ff00af8e0 0x7f6ff0134810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:57845/0 (socket says 172.21.15.100:57845) 2024-03-29T11:39:30.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.648+0000 7f6ff546b640 1 -- 172.21.15.100:0/3242193128 learned_addr learned my addr 172.21.15.100:0/3242193128 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:30.651 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.650+0000 7f6fe5ffb640 10 client.?.objecter ms_handle_connect 0x7f6ff00af8e0 2024-03-29T11:39:30.651 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.650+0000 7f6fe5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:30.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141367203 0 0) 0x7f6ff006e6b0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:30.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:30.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:30.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:39:30.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd4004eb0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.652 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 client.?.objecter ms_handle_connect 0x7f6ff013ef30 2024-03-29T11:39:30.653 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:30.653 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3163784642 0 0) 0x7f6fd4004eb0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.653 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ff006e6b0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.653 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6fec001960 con 0x7f6ff00af8e0 2024-03-29T11:39:30.653 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:30.653 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:30.653 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:30.653 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:30.653 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:30.653 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539971423 0 0) 0x7f6ff0070d10 con 0x7f6ff013ef30 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:30.654 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:39:30.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.651+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd4004eb0 con 0x7f6ff013ef30 2024-03-29T11:39:30.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 10 client.?.objecter ms_handle_connect 0x7f6ff00b0860 2024-03-29T11:39:30.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:30.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 231020384 0 0) 0x7f6ff006e6b0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 >> v1:172.21.15.191:6789/0 conn(0x7f6ff013ef30 legacy=0x7f6ff006f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:30.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 >> v1:172.21.15.100:6789/0 conn(0x7f6ff00b0860 legacy=0x7f6ff006ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:30.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:30.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 1 monclient: found mon.c 2024-03-29T11:39:30.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:30.655 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:30.656 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff006e840 con 0x7f6ff00af8e0 2024-03-29T11:39:30.656 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:30.656 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:30.656 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.653+0000 7f6fe5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:30.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.654+0000 7f6ff76cf640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:39:30.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.654+0000 7f6ff76cf640 10 monclient: _renew_subs 2024-03-29T11:39:30.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.654+0000 7f6ff76cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.654+0000 7f6ff76cf640 1 -- 172.21.15.100:0/3242193128 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff00714a0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6ff76cf640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6ff76cf640 10 monclient: _renew_subs 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6ff76cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6ff76cf640 1 -- 172.21.15.100:0/3242193128 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff0135100 con 0x7f6ff00af8e0 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6ff76cf640 1 librados: init done 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fec003610 con 0x7f6ff00af8e0 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6fe5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6fec0039e0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6fe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6fe5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6fe5ffb640 10 monclient: dump: 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:30.658 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:30.659 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:30.659 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:30.659 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:30.659 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:30.659 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:30.659 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:30.659 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:30.659 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:30.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.655+0000 7f6fe77fe640 4 set_mon_vals no callback set 2024-03-29T11:39:30.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.657+0000 7f6ff76cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:30.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.657+0000 7f6ff76cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:30.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.657+0000 7f6ff76cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:30.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.657+0000 7f6ff76cf640 1 -- 172.21.15.100:0/3242193128 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0005080 con 0x7f6ff00af8e0 2024-03-29T11:39:30.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.658+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f6fec003bc0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.659+0000 7f6ff4c6a640 10 monclient: get_auth_request con 0x7f6fd406ddd0 auth_method 0 2024-03-29T11:39:30.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.659+0000 7f6fe5ffb640 10 client.4349.objecter ms_handle_connect 0x7f6fd406ddd0 2024-03-29T11:39:30.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.659+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f6fec003800 con 0x7f6ff00af8e0 2024-03-29T11:39:30.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.659+0000 7f6fe5ffb640 10 client.4349.objecter ms_dispatch 0x7f6ff0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:30.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.659+0000 7f6fe5ffb640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:30.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.659+0000 7f6fe5ffb640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:30.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.659+0000 7f6fe5ffb640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T11:39:30.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.673+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fec0509b0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.673+0000 7f6fe5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:30.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.673+0000 7f6fe5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.997+0000 7f6ff76cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.997+0000 7f6ff76cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.997+0000 7f6ff76cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.997+0000 7f6ff76cf640 1 -- 172.21.15.100:0/3242193128 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fd0005330 con 0x7f6ff00af8e0 2024-03-29T11:39:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.998+0000 7f6fe5ffb640 1 -- 172.21.15.100:0/3242193128 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f6fec0509b0 con 0x7f6ff00af8e0 2024-03-29T11:39:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.998+0000 7f6fe5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:30.998+0000 7f6fe5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:31.000 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:39:31.000 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:31.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.002+0000 7f6ff76cf640 10 librados: watch_flush enter 2024-03-29T11:39:31.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.002+0000 7f6ff76cf640 10 librados: watch_flush exit 2024-03-29T11:39:31.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.002+0000 7f6ff76cf640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T11:39:31.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.002+0000 7f6ff76cf640 10 client.4349.objecter successfully canceled tick 2024-03-29T11:39:31.004 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.002+0000 7f6ff76cf640 1 -- 172.21.15.100:0/3242193128 >> v1:172.21.15.100:6816/3220289657 conn(0x7f6fd406ddd0 legacy=0x7f6fd4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:31.004 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.002+0000 7f6ff76cf640 10 monclient: shutdown 2024-03-29T11:39:31.004 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.002+0000 7f6ff76cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:31.004 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.002+0000 7f6ff76cf640 1 -- 172.21.15.100:0/3242193128 >> v1:172.21.15.100:6790/0 conn(0x7f6ff00af8e0 legacy=0x7f6ff0134810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:31.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.004+0000 7f6ff76cf640 1 -- 172.21.15.100:0/3242193128 shutdown_connections 2024-03-29T11:39:31.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.005+0000 7f6ff76cf640 1 -- 172.21.15.100:0/3242193128 >> 172.21.15.100:0/3242193128 conn(0x7f6ff006b660 msgr2=0x7f6ff013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:31.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.005+0000 7f6ff76cf640 1 -- 172.21.15.100:0/3242193128 shutdown_connections 2024-03-29T11:39:31.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.005+0000 7f6ff76cf640 1 -- 172.21.15.100:0/3242193128 wait complete. 2024-03-29T11:39:31.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:31.005+0000 7f6ff76cf640 1 librados: shutdown 2024-03-29T11:39:31.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:37.830 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:37.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.984+0000 7fcc7d835640 10 monclient: get_monmap_and_config 2024-03-29T11:39:37.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.984+0000 7fcc7d835640 10 monclient: build_initial_monmap 2024-03-29T11:39:37.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.985+0000 7fcc7d835640 10 monclient: monmap: 2024-03-29T11:39:37.987 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:37.987 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:37.987 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:39:37.986022+0000 2024-03-29T11:39:37.987 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:39:37.986022+0000 2024-03-29T11:39:37.987 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:37.987 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:37.987 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:37.987 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:39:37.987 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:39:37.987 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:37.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.986+0000 7fcc7d835640 1 Processor -- start 2024-03-29T11:39:37.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.986+0000 7fcc7d835640 1 -- start start 2024-03-29T11:39:37.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.986+0000 7fcc7d835640 10 monclient: init 2024-03-29T11:39:37.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.987+0000 7fcc7d835640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:37.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.987+0000 7fcc7d835640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:37.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.987+0000 7fcc7d835640 10 monclient(hunting): picked mon.noname-c con 0x7fcc780b1f60 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:37.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.987+0000 7fcc7d835640 10 monclient(hunting): picked mon.noname-a con 0x7fcc78071eb0 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:37.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.987+0000 7fcc7d835640 10 monclient(hunting): picked mon.noname-b con 0x7fcc78143d10 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:37.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.987+0000 7fcc7d835640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7806ef80 con 0x7fcc78071eb0 2024-03-29T11:39:37.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.987+0000 7fcc7d835640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7806f260 con 0x7fcc780b1f60 2024-03-29T11:39:37.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.987+0000 7fcc7d835640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc780b40c0 con 0x7fcc78143d10 2024-03-29T11:39:37.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.987+0000 7fcc7d835640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:37.989 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.987+0000 7fcc7d835640 10 monclient(hunting): authenticate will time out at 1098.182498s 2024-03-29T11:39:37.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.991+0000 7fcc767fc640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fcc78071eb0 0x7fcc78146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:33931/0 (socket says 172.21.15.100:33931) 2024-03-29T11:39:37.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.991+0000 7fcc767fc640 1 -- 172.21.15.100:0/3417938980 learned_addr learned my addr 172.21.15.100:0/3417938980 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:37.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 1 -- 172.21.15.100:0/3417938980 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931005051 0 0) 0x7fcc7806f260 con 0x7fcc780b1f60 2024-03-29T11:39:37.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:37.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:37.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:37.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:39:37.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 1 -- 172.21.15.100:0/3417938980 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc60003520 con 0x7fcc780b1f60 2024-03-29T11:39:37.997 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 1 -- 172.21.15.100:0/3417938980 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201806449 0 0) 0x7fcc780b40c0 con 0x7fcc78143d10 2024-03-29T11:39:37.997 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:37.997 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:37.997 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:37.997 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:39:37.997 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 1 -- 172.21.15.100:0/3417938980 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc7806f260 con 0x7fcc78143d10 2024-03-29T11:39:37.997 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.995+0000 7fcc75ffb640 1 -- 172.21.15.100:0/3417938980 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894462240 0 0) 0x7fcc60003520 con 0x7fcc780b1f60 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 1 -- 172.21.15.100:0/3417938980 >> v1:172.21.15.100:6790/0 conn(0x7fcc78143d10 legacy=0x7fcc78146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 1 -- 172.21.15.100:0/3417938980 >> v1:172.21.15.100:6789/0 conn(0x7fcc78071eb0 legacy=0x7fcc78146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 1 monclient: found mon.noname-c 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 1 -- 172.21.15.100:0/3417938980 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc78155d70 con 0x7fcc780b1f60 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 1 -- 172.21.15.100:0/3417938980 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fcc68001610 con 0x7fcc780b1f60 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:37.998 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 10 monclient: dump: 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc7d835640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:39:37.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc7d835640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:38.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 1 -- 172.21.15.100:0/3417938980 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc68001830 con 0x7fcc780b1f60 2024-03-29T11:39:38.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:38.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc7d835640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:38.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.996+0000 7fcc7d835640 4 set_mon_vals no callback set 2024-03-29T11:39:38.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.997+0000 7fcc7d835640 10 monclient: shutdown 2024-03-29T11:39:38.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.997+0000 7fcc7d835640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:38.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.997+0000 7fcc7d835640 1 -- 172.21.15.100:0/3417938980 >> v1:172.21.15.191:6789/0 conn(0x7fcc780b1f60 legacy=0x7fcc78143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:38.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.997+0000 7fcc7d835640 1 -- 172.21.15.100:0/3417938980 shutdown_connections 2024-03-29T11:39:38.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.997+0000 7fcc7d835640 1 -- 172.21.15.100:0/3417938980 >> 172.21.15.100:0/3417938980 conn(0x7fcc78066a00 msgr2=0x7fcc78066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:38.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.998+0000 7fcc7d835640 1 -- 172.21.15.100:0/3417938980 shutdown_connections 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.998+0000 7fcc7d835640 1 -- 172.21.15.100:0/3417938980 wait complete. 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.999+0000 7fcc7d835640 10 monclient: build_initial_monmap 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.999+0000 7fcc7d835640 10 monclient: monmap: 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:38.002 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.999+0000 7fcc7d835640 1 librados: starting msgr at 2024-03-29T11:39:38.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.999+0000 7fcc7d835640 1 librados: starting objecter 2024-03-29T11:39:38.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:37.999+0000 7fcc7d835640 1 Processor -- start 2024-03-29T11:39:38.004 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.000+0000 7fcc7d835640 1 -- start start 2024-03-29T11:39:38.004 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.000+0000 7fcc7d835640 1 librados: setting wanted keys 2024-03-29T11:39:38.004 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.000+0000 7fcc7d835640 1 librados: calling monclient init 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.000+0000 7fcc7d835640 10 monclient: init 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.001+0000 7fcc7d835640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.001+0000 7fcc7d835640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.001+0000 7fcc7d835640 10 monclient(hunting): picked mon.noname-b con 0x7fcc780b1f60 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.001+0000 7fcc7d835640 10 monclient(hunting): picked mon.noname-c con 0x7fcc78143d10 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.001+0000 7fcc7d835640 10 monclient(hunting): picked mon.noname-a con 0x7fcc780aa410 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.001+0000 7fcc7d835640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc781e4d60 con 0x7fcc780aa410 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.001+0000 7fcc7d835640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc780aca70 con 0x7fcc780b1f60 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.001+0000 7fcc7d835640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc781458d0 con 0x7fcc78143d10 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.001+0000 7fcc7d835640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:38.005 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.001+0000 7fcc7d835640 10 monclient(hunting): authenticate will time out at 1098.196702s 2024-03-29T11:39:38.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.004+0000 7fcc767fc640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fcc78143d10 0x7fcc781e4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:48632/0 (socket says 172.21.15.100:48632) 2024-03-29T11:39:38.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.004+0000 7fcc767fc640 1 -- 172.21.15.100:0/4062573916 learned_addr learned my addr 172.21.15.100:0/4062573916 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:38.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.005+0000 7fcc5ffff640 10 client.?.objecter ms_handle_connect 0x7fcc78143d10 2024-03-29T11:39:38.006 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.005+0000 7fcc5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:38.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.005+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523479720 0 0) 0x7fcc781458d0 con 0x7fcc78143d10 2024-03-29T11:39:38.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.005+0000 7fcc5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:38.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.005+0000 7fcc5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.005+0000 7fcc5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.005+0000 7fcc5ffff640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.005+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc64004eb0 con 0x7fcc78143d10 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.006+0000 7fcc5ffff640 10 client.?.objecter ms_handle_connect 0x7fcc780b1f60 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.006+0000 7fcc5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.006+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187576066 0 0) 0x7fcc780aca70 con 0x7fcc780b1f60 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.006+0000 7fcc5ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.006+0000 7fcc5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.006+0000 7fcc5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.006+0000 7fcc5ffff640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:39:38.008 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.006+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc781458d0 con 0x7fcc780b1f60 2024-03-29T11:39:38.009 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3744448141 0 0) 0x7fcc781458d0 con 0x7fcc780b1f60 2024-03-29T11:39:38.009 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc64005490 con 0x7fcc780b1f60 2024-03-29T11:39:38.009 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fcc680010e0 con 0x7fcc780b1f60 2024-03-29T11:39:38.009 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 10 monclient(hunting): dump: 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:38.010 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 10 client.?.objecter ms_handle_connect 0x7fcc780aa410 2024-03-29T11:39:38.011 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:38.011 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596013742 0 0) 0x7fcc64004eb0 con 0x7fcc78143d10 2024-03-29T11:39:38.011 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc64005b60 con 0x7fcc78143d10 2024-03-29T11:39:38.011 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fcc70001cb0 con 0x7fcc78143d10 2024-03-29T11:39:38.011 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 10 monclient(hunting): dump: 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:38.012 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4079259883 0 0) 0x7fcc64005490 con 0x7fcc780b1f60 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 >> v1:172.21.15.100:6790/0 conn(0x7fcc78143d10 legacy=0x7fcc781e4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.007+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 >> v1:172.21.15.100:6789/0 conn(0x7fcc780aa410 legacy=0x7fcc7813fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc5ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc5ffff640 1 monclient: found mon.b 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc5ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc78145a60 con 0x7fcc780b1f60 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc5ffff640 10 monclient: _finish_auth 0 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc5ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc7d835640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc7d835640 10 monclient: _renew_subs 2024-03-29T11:39:38.013 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc7d835640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc7d835640 1 -- 172.21.15.100:0/4062573916 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc78140360 con 0x7fcc780b1f60 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc7d835640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc7d835640 10 monclient: _renew_subs 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc7d835640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc7d835640 1 -- 172.21.15.100:0/4062573916 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc781f0770 con 0x7fcc780b1f60 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.009+0000 7fcc7d835640 1 librados: init done 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.010+0000 7fcc5dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.010+0000 7fcc5dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.010+0000 7fcc5dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.010+0000 7fcc5dffb640 1 -- 172.21.15.100:0/4062573916 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc780ac890 con 0x7fcc780b1f60 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.010+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc68001a20 con 0x7fcc780b1f60 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.010+0000 7fcc5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.011+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fcc680010e0 con 0x7fcc780b1f60 2024-03-29T11:39:38.014 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.011+0000 7fcc5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.011+0000 7fcc5ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.011+0000 7fcc5ffff640 10 monclient: dump: 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:38.015 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:38.016 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.011+0000 7fcc757fa640 4 set_mon_vals no callback set 2024-03-29T11:39:38.016 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.012+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fcc680198b0 con 0x7fcc780b1f60 2024-03-29T11:39:38.017 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.013+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fcc68053100 con 0x7fcc780b1f60 2024-03-29T11:39:38.017 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.013+0000 7fcc5ffff640 10 client.4342.objecter ms_dispatch 0x7fcc78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:38.017 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.013+0000 7fcc5ffff640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:38.017 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.013+0000 7fcc5ffff640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:38.017 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.013+0000 7fcc5ffff640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-29T11:39:38.017 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.013+0000 7fcc767fc640 10 monclient: get_auth_request con 0x7fcc6406e590 auth_method 0 2024-03-29T11:39:38.017 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.014+0000 7fcc5ffff640 10 client.4342.objecter ms_handle_connect 0x7fcc6406e590 2024-03-29T11:39:38.019 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.017+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc6804f3a0 con 0x7fcc780b1f60 2024-03-29T11:39:38.019 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.017+0000 7fcc5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:38.019 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.017+0000 7fcc5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:38.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.343+0000 7fcc5dffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:38.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.343+0000 7fcc5dffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:38.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.343+0000 7fcc5dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:39:38.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.343+0000 7fcc5dffb640 1 -- 172.21.15.100:0/4062573916 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc781e3f20 con 0x7fcc780b1f60 2024-03-29T11:39:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.346+0000 7fcc5ffff640 1 -- 172.21.15.100:0/4062573916 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fcc6804f3a0 con 0x7fcc780b1f60 2024-03-29T11:39:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.346+0000 7fcc5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.346+0000 7fcc5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:38.349 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:39:38.349 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:38.358 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.352+0000 7fcc5dffb640 10 librados: watch_flush enter 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.353+0000 7fcc5dffb640 10 librados: watch_flush exit 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 10 client.4342.objecter successfully canceled tick 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 1 -- 172.21.15.100:0/4062573916 >> v1:172.21.15.100:6816/3220289657 conn(0x7fcc6406e590 legacy=0x7fcc64070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 10 monclient: shutdown 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 1 -- 172.21.15.100:0/4062573916 >> v1:172.21.15.191:6789/0 conn(0x7fcc780b1f60 legacy=0x7fcc780ab100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 1 -- 172.21.15.100:0/4062573916 shutdown_connections 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 1 -- 172.21.15.100:0/4062573916 >> 172.21.15.100:0/4062573916 conn(0x7fcc78066a00 msgr2=0x7fcc78066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 1 -- 172.21.15.100:0/4062573916 shutdown_connections 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 1 -- 172.21.15.100:0/4062573916 wait complete. 2024-03-29T11:39:38.359 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:38.354+0000 7fcc5dffb640 1 librados: shutdown 2024-03-29T11:39:38.376 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:45.183 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:45.319 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.318+0000 7f354cce4640 10 monclient: get_monmap_and_config 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.318+0000 7f354cce4640 10 monclient: build_initial_monmap 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.318+0000 7f354cce4640 10 monclient: monmap: 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:39:45.319143+0000 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:39:45.319143+0000 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:39:45.320 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:45.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.318+0000 7f354cce4640 1 Processor -- start 2024-03-29T11:39:45.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.319+0000 7f354cce4640 1 -- start start 2024-03-29T11:39:45.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.319+0000 7f354cce4640 10 monclient: init 2024-03-29T11:39:45.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f354cce4640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:45.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f354cce4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:45.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f354cce4640 10 monclient(hunting): picked mon.noname-a con 0x7f35481457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:45.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f354cce4640 10 monclient(hunting): picked mon.noname-b con 0x7f354813f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:45.323 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f354cce4640 10 monclient(hunting): picked mon.noname-c con 0x7f35480a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f354cce4640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35480aaef0 con 0x7f35481457f0 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f354cce4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3548145bd0 con 0x7f35480a9d70 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f354cce4640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3548145eb0 con 0x7f354813f650 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f354cce4640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f354cce4640 10 monclient(hunting): authenticate will time out at 1105.515231s 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f3545d9b640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f354813f650 0x7f35480b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:51566/0 (socket says 172.21.15.100:51566) 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.320+0000 7f3545d9b640 1 -- 172.21.15.100:0/2141291342 learned_addr learned my addr 172.21.15.100:0/2141291342 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.321+0000 7f354559a640 1 -- 172.21.15.100:0/2141291342 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262959549 0 0) 0x7f3548145eb0 con 0x7f354813f650 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.321+0000 7f354559a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.321+0000 7f354559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.321+0000 7f354559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.321+0000 7f354559a640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:39:45.324 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.321+0000 7f354559a640 1 -- 172.21.15.100:0/2141291342 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3530003520 con 0x7f354813f650 2024-03-29T11:39:45.325 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.321+0000 7f354559a640 1 -- 172.21.15.100:0/2141291342 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1456731622 0 0) 0x7f3530003520 con 0x7f354813f650 2024-03-29T11:39:45.325 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.321+0000 7f354559a640 1 -- 172.21.15.100:0/2141291342 >> v1:172.21.15.191:6789/0 conn(0x7f35480a9d70 legacy=0x7f354814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 1 -- 172.21.15.100:0/2141291342 >> v1:172.21.15.100:6789/0 conn(0x7f35481457f0 legacy=0x7f354813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 10 monclient: _finish_hunting 0 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 1 monclient: found mon.noname-b 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 1 -- 172.21.15.100:0/2141291342 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3548146340 con 0x7f354813f650 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 10 monclient: _finish_auth 0 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 10 monclient: _check_auth_tickets 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 1 -- 172.21.15.100:0/2141291342 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f353c001690 con 0x7f354813f650 2024-03-29T11:39:45.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354559a640 10 monclient: dump: 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354cce4640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.322+0000 7f354cce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:45.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.323+0000 7f354559a640 1 -- 172.21.15.100:0/2141291342 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f353c0018f0 con 0x7f354813f650 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.323+0000 7f354559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.323+0000 7f354559a640 1 -- 172.21.15.100:0/2141291342 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f353c001d00 con 0x7f354813f650 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.323+0000 7f354559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.323+0000 7f354559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.323+0000 7f354559a640 10 monclient: dump: 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:45.338 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.323+0000 7f354cce4640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.323+0000 7f354cce4640 4 set_mon_vals no callback set 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.324+0000 7f354cce4640 10 monclient: shutdown 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.324+0000 7f354cce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.324+0000 7f354cce4640 1 -- 172.21.15.100:0/2141291342 >> v1:172.21.15.100:6790/0 conn(0x7f354813f650 legacy=0x7f35480b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.324+0000 7f354cce4640 1 -- 172.21.15.100:0/2141291342 shutdown_connections 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.324+0000 7f354cce4640 1 -- 172.21.15.100:0/2141291342 >> 172.21.15.100:0/2141291342 conn(0x7f3548066a00 msgr2=0x7f3548066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.324+0000 7f354cce4640 1 -- 172.21.15.100:0/2141291342 shutdown_connections 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.324+0000 7f354cce4640 1 -- 172.21.15.100:0/2141291342 wait complete. 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.325+0000 7f354cce4640 10 monclient: build_initial_monmap 2024-03-29T11:39:45.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.325+0000 7f354cce4640 10 monclient: monmap: 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.326+0000 7f354cce4640 1 librados: starting msgr at 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.326+0000 7f354cce4640 1 librados: starting objecter 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.326+0000 7f354cce4640 1 Processor -- start 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.326+0000 7f354cce4640 1 -- start start 2024-03-29T11:39:45.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.326+0000 7f354cce4640 1 librados: setting wanted keys 2024-03-29T11:39:45.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.326+0000 7f354cce4640 1 librados: calling monclient init 2024-03-29T11:39:45.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.326+0000 7f354cce4640 10 monclient: init 2024-03-29T11:39:45.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f354cce4640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:45.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f354cce4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:45.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f354cce4640 10 monclient(hunting): picked mon.noname-b con 0x7f35480a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:45.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f354cce4640 10 monclient(hunting): picked mon.noname-a con 0x7f354813f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:45.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f354cce4640 10 monclient(hunting): picked mon.noname-c con 0x7f35481457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:45.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f354cce4640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354813ac90 con 0x7f354813f650 2024-03-29T11:39:45.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f354cce4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354813aec0 con 0x7f35480a9d70 2024-03-29T11:39:45.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f354cce4640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354813a2b0 con 0x7f35481457f0 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f354cce4640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f354cce4640 10 monclient(hunting): authenticate will time out at 1105.522595s 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f3546d9d640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f35481457f0 0x7f35481350e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:51808/0 (socket says 172.21.15.100:51808) 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f3546d9d640 1 -- 172.21.15.100:0/206389993 learned_addr learned my addr 172.21.15.100:0/206389993 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f3536ffd640 10 client.?.objecter ms_handle_connect 0x7f35481457f0 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.327+0000 7f3536ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 10 client.?.objecter ms_handle_connect 0x7f354813f650 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853972010 0 0) 0x7f354813a2b0 con 0x7f35481457f0 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:45.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3528004eb0 con 0x7f35481457f0 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339109083 0 0) 0x7f354813ac90 con 0x7f354813f650 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f354813a2b0 con 0x7f354813f650 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3885186492 0 0) 0x7f3528004eb0 con 0x7f35481457f0 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f354813ac90 con 0x7f35481457f0 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.328+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3540001b70 con 0x7f35481457f0 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:45.343 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2367021304 0 0) 0x7f354813a2b0 con 0x7f354813f650 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3528004eb0 con 0x7f354813f650 2024-03-29T11:39:45.344 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f353c0014b0 con 0x7f354813f650 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:45.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2209216702 0 0) 0x7f354813ac90 con 0x7f35481457f0 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 >> v1:172.21.15.191:6789/0 conn(0x7f35480a9d70 legacy=0x7f3548142630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 >> v1:172.21.15.100:6789/0 conn(0x7f354813f650 legacy=0x7f354813b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 1 monclient: found mon.c 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.329+0000 7f3536ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f3536ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354813a590 con 0x7f35481457f0 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f3536ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f3536ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f3536ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:45.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f354cce4640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f354cce4640 10 monclient: _renew_subs 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f354cce4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f354cce4640 1 -- 172.21.15.100:0/206389993 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354813a870 con 0x7f35481457f0 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f354cce4640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f354cce4640 10 monclient: _renew_subs 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f354cce4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f354cce4640 1 -- 172.21.15.100:0/206389993 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35481359d0 con 0x7f35481457f0 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f354cce4640 1 librados: init done 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3540001cd0 con 0x7f35481457f0 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f3536ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3540004e20 con 0x7f35481457f0 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.330+0000 7f3544d99640 4 set_mon_vals no callback set 2024-03-29T11:39:45.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.331+0000 7f3536ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.331+0000 7f3536ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.331+0000 7f3536ffd640 10 monclient: dump: 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:45.348 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.331+0000 7f354cce4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.332+0000 7f354cce4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.332+0000 7f354cce4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.332+0000 7f354cce4640 1 -- 172.21.15.100:0/206389993 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f352c005080 con 0x7f35481457f0 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.333+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f354001adf0 con 0x7f35481457f0 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.333+0000 7f354659c640 10 monclient: get_auth_request con 0x7f352806df90 auth_method 0 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.334+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f3540084d80 con 0x7f35481457f0 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.334+0000 7f3536ffd640 10 client.4361.objecter ms_dispatch 0x7f3548000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.334+0000 7f3536ffd640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.334+0000 7f3536ffd640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.334+0000 7f3536ffd640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-29T11:39:45.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.334+0000 7f3536ffd640 10 client.4361.objecter ms_handle_connect 0x7f352806df90 2024-03-29T11:39:45.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.340+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3540050660 con 0x7f35481457f0 2024-03-29T11:39:45.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.340+0000 7f3536ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:45.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.340+0000 7f3536ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:45.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.699+0000 7f354cce4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:45.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.699+0000 7f354cce4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:45.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.699+0000 7f354cce4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:45.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.699+0000 7f354cce4640 1 -- 172.21.15.100:0/206389993 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f352c005330 con 0x7f35481457f0 2024-03-29T11:39:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.701+0000 7f3536ffd640 1 -- 172.21.15.100:0/206389993 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f3540052590 con 0x7f35481457f0 2024-03-29T11:39:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.701+0000 7f3536ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:45.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.701+0000 7f3536ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:45.705 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:39:45.705 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:45.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.707+0000 7f354cce4640 10 librados: watch_flush enter 2024-03-29T11:39:45.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.707+0000 7f354cce4640 10 librados: watch_flush exit 2024-03-29T11:39:45.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.707+0000 7f354cce4640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-29T11:39:45.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.707+0000 7f354cce4640 10 client.4361.objecter successfully canceled tick 2024-03-29T11:39:45.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.708+0000 7f354cce4640 1 -- 172.21.15.100:0/206389993 >> v1:172.21.15.100:6816/3220289657 conn(0x7f352806df90 legacy=0x7f3528070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:45.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.708+0000 7f354cce4640 10 monclient: shutdown 2024-03-29T11:39:45.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.708+0000 7f354cce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:45.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.708+0000 7f354cce4640 1 -- 172.21.15.100:0/206389993 >> v1:172.21.15.100:6790/0 conn(0x7f35481457f0 legacy=0x7f35481350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:45.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.708+0000 7f354cce4640 1 -- 172.21.15.100:0/206389993 shutdown_connections 2024-03-29T11:39:45.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.708+0000 7f354cce4640 1 -- 172.21.15.100:0/206389993 >> 172.21.15.100:0/206389993 conn(0x7f3548066a00 msgr2=0x7f35480b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:45.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.708+0000 7f354cce4640 1 -- 172.21.15.100:0/206389993 shutdown_connections 2024-03-29T11:39:45.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.708+0000 7f354cce4640 1 -- 172.21.15.100:0/206389993 wait complete. 2024-03-29T11:39:45.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:45.709+0000 7f354cce4640 1 librados: shutdown 2024-03-29T11:39:45.726 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:52.534 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:52.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.686+0000 7f368c6f1640 10 monclient: get_monmap_and_config 2024-03-29T11:39:52.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.686+0000 7f368c6f1640 10 monclient: build_initial_monmap 2024-03-29T11:39:52.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.686+0000 7f368c6f1640 10 monclient: monmap: 2024-03-29T11:39:52.688 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:52.688 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:52.688 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:39:52.687769+0000 2024-03-29T11:39:52.688 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:39:52.687769+0000 2024-03-29T11:39:52.688 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:52.688 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:52.689 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:52.689 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:39:52.689 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:39:52.689 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:52.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.687+0000 7f368c6f1640 1 Processor -- start 2024-03-29T11:39:52.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.687+0000 7f368c6f1640 1 -- start start 2024-03-29T11:39:52.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.687+0000 7f368c6f1640 10 monclient: init 2024-03-29T11:39:52.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.688+0000 7f368c6f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:52.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.688+0000 7f368c6f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:52.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.688+0000 7f368c6f1640 10 monclient(hunting): picked mon.noname-b con 0x7f36841457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:52.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.688+0000 7f368c6f1640 10 monclient(hunting): picked mon.noname-c con 0x7f368413f650 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:52.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368c6f1640 10 monclient(hunting): picked mon.noname-a con 0x7f36840a9d70 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:52.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368c6f1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36840aaef0 con 0x7f36840a9d70 2024-03-29T11:39:52.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368ac8e640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f36840a9d70 0x7f368414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:33118/0 (socket says 172.21.15.100:33118) 2024-03-29T11:39:52.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368a48d640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f36841457f0 0x7f368413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:47686/0 (socket says 172.21.15.100:47686) 2024-03-29T11:39:52.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368ac8e640 1 -- 172.21.15.100:0/3429885688 learned_addr learned my addr 172.21.15.100:0/3429885688 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:52.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368c6f1640 1 -- 172.21.15.100:0/3429885688 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3684145bd0 con 0x7f368413f650 2024-03-29T11:39:52.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368c6f1640 1 -- 172.21.15.100:0/3429885688 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3684145eb0 con 0x7f36841457f0 2024-03-29T11:39:52.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368c6f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:52.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368c6f1640 10 monclient(hunting): authenticate will time out at 1112.884308s 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489552079 0 0) 0x7f3684145eb0 con 0x7f36841457f0 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368948b640 10 monclient(hunting): _negotiate 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368948b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368948b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368948b640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.689+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3670003520 con 0x7f36841457f0 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791262471 0 0) 0x7f36840aaef0 con 0x7f36840a9d70 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient(hunting): _negotiate 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3684145eb0 con 0x7f36840a9d70 2024-03-29T11:39:52.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306789028 0 0) 0x7f3684145bd0 con 0x7f368413f650 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient(hunting): _negotiate 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36840aaef0 con 0x7f368413f650 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2650644074 0 0) 0x7f3684145eb0 con 0x7f36840a9d70 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 >> v1:172.21.15.100:6790/0 conn(0x7f36841457f0 legacy=0x7f368413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 >> v1:172.21.15.191:6789/0 conn(0x7f368413f650 legacy=0x7f36840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient: _finish_hunting 0 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 1 monclient: found mon.noname-a 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3684146340 con 0x7f36840a9d70 2024-03-29T11:39:52.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient: _finish_auth 0 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient: _check_auth_tickets 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368c6f1640 5 monclient: authenticate success, global_id 4347 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f367c0019f0 con 0x7f36840a9d70 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368c6f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368948b640 10 monclient: dump: 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:52.697 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.690+0000 7f368c6f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.691+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f367c001c10 con 0x7f36840a9d70 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.691+0000 7f368948b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.691+0000 7f368948b640 1 -- 172.21.15.100:0/3429885688 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f367c001fa0 con 0x7f36840a9d70 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.691+0000 7f368c6f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.691+0000 7f368c6f1640 4 set_mon_vals no callback set 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.691+0000 7f368948b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.691+0000 7f368948b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.691+0000 7f368948b640 10 monclient: dump: 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:52.698 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.692+0000 7f368c6f1640 10 monclient: shutdown 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.692+0000 7f368c6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.692+0000 7f368c6f1640 1 -- 172.21.15.100:0/3429885688 >> v1:172.21.15.100:6789/0 conn(0x7f36840a9d70 legacy=0x7f368414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.692+0000 7f368c6f1640 1 -- 172.21.15.100:0/3429885688 shutdown_connections 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.692+0000 7f368c6f1640 1 -- 172.21.15.100:0/3429885688 >> 172.21.15.100:0/3429885688 conn(0x7f3684066a00 msgr2=0x7f3684066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.692+0000 7f368c6f1640 1 -- 172.21.15.100:0/3429885688 shutdown_connections 2024-03-29T11:39:52.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.693+0000 7f368c6f1640 1 -- 172.21.15.100:0/3429885688 wait complete. 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.693+0000 7f368c6f1640 10 monclient: build_initial_monmap 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.693+0000 7f368c6f1640 10 monclient: monmap: 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:39:52.700 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:52.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.694+0000 7f368c6f1640 1 librados: starting msgr at 2024-03-29T11:39:52.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.694+0000 7f368c6f1640 1 librados: starting objecter 2024-03-29T11:39:52.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.694+0000 7f368c6f1640 1 Processor -- start 2024-03-29T11:39:52.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.694+0000 7f368c6f1640 1 -- start start 2024-03-29T11:39:52.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.694+0000 7f368c6f1640 1 librados: setting wanted keys 2024-03-29T11:39:52.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.694+0000 7f368c6f1640 1 librados: calling monclient init 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.694+0000 7f368c6f1640 10 monclient: init 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.695+0000 7f368c6f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.695+0000 7f368c6f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f368c6f1640 10 monclient(hunting): picked mon.noname-b con 0x7f36840a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f368c6f1640 10 monclient(hunting): picked mon.noname-a con 0x7f368413f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f368c6f1640 10 monclient(hunting): picked mon.noname-c con 0x7f36841457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f3689c8c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f368413f650 0x7f3684140660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:33364/0 (socket says 172.21.15.100:33364) 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f3689c8c640 1 -- 172.21.15.100:0/2901147174 learned_addr learned my addr 172.21.15.100:0/2901147174 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3684142d00 con 0x7f368413f650 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36841cbf30 con 0x7f36840a9d70 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f367affd640 10 client.?.objecter ms_handle_connect 0x7f36841457f0 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f367affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:52.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f367affd640 10 client.?.objecter ms_handle_connect 0x7f368413f650 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f367affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36841cc210 con 0x7f36841457f0 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f368c6f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f368c6f1640 10 monclient(hunting): authenticate will time out at 1112.891803s 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.696+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976834916 0 0) 0x7f36841cc210 con 0x7f36841457f0 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3668004eb0 con 0x7f36841457f0 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718124443 0 0) 0x7f3684142d00 con 0x7f368413f650 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:39:52.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36841cc210 con 0x7f368413f650 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1431735498 0 0) 0x7f3668004eb0 con 0x7f36841457f0 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3684142d00 con 0x7f36841457f0 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f367c001930 con 0x7f36841457f0 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): dump: 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:52.707 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2943990648 0 0) 0x7f36841cc210 con 0x7f368413f650 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3668004eb0 con 0x7f368413f650 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3674001970 con 0x7f368413f650 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 10 monclient(hunting): dump: 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:52.708 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.697+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3188200132 0 0) 0x7f3684142d00 con 0x7f36841457f0 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 >> v1:172.21.15.191:6789/0 conn(0x7f36840a9d70 legacy=0x7f368413ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 >> v1:172.21.15.100:6789/0 conn(0x7f368413f650 legacy=0x7f3684140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 1 monclient: found mon.c 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36841cc4f0 con 0x7f36841457f0 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 10 monclient: _finish_auth 0 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:52.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f368c6f1640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f368c6f1640 10 monclient: _renew_subs 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f368c6f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36841426c0 con 0x7f36841457f0 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f368c6f1640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f368c6f1640 10 monclient: _renew_subs 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f368c6f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3684144a90 con 0x7f36841457f0 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f368c6f1640 1 librados: init done 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f367c004e10 con 0x7f36841457f0 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f367c005120 con 0x7f36841457f0 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:52.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.698+0000 7f367affd640 10 monclient: dump: 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.699+0000 7f367bfff640 4 set_mon_vals no callback set 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.700+0000 7f368c6f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:52.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.700+0000 7f368c6f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:52.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.700+0000 7f368c6f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:52.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.700+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366c005080 con 0x7f36841457f0 2024-03-29T11:39:52.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.701+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f367c01b0f0 con 0x7f36841457f0 2024-03-29T11:39:52.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.701+0000 7f368a48d640 10 monclient: get_auth_request con 0x7f366806de80 auth_method 0 2024-03-29T11:39:52.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.702+0000 7f367affd640 10 client.4367.objecter ms_handle_connect 0x7f366806de80 2024-03-29T11:39:52.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.702+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f367c084e30 con 0x7f36841457f0 2024-03-29T11:39:52.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.702+0000 7f367affd640 10 client.4367.objecter ms_dispatch 0x7f3684000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:52.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.702+0000 7f367affd640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:52.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.702+0000 7f367affd640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:52.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.702+0000 7f367affd640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-29T11:39:52.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.710+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f367c050710 con 0x7f36841457f0 2024-03-29T11:39:52.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.710+0000 7f367affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:52.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:52.710+0000 7f367affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:53.118 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.116+0000 7f368c6f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:53.118 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.116+0000 7f368c6f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:53.118 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.116+0000 7f368c6f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:39:53.118 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.116+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f366c005330 con 0x7f36841457f0 2024-03-29T11:39:53.118 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.117+0000 7f367affd640 1 -- 172.21.15.100:0/2901147174 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f367c052640 con 0x7f36841457f0 2024-03-29T11:39:53.119 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.117+0000 7f367affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:53.119 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.117+0000 7f367affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:53.119 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:39:53.119 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:53.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 10 librados: watch_flush enter 2024-03-29T11:39:53.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 10 librados: watch_flush exit 2024-03-29T11:39:53.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-29T11:39:53.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 10 client.4367.objecter successfully canceled tick 2024-03-29T11:39:53.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 >> v1:172.21.15.100:6816/3220289657 conn(0x7f366806de80 legacy=0x7f3668070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:53.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 10 monclient: shutdown 2024-03-29T11:39:53.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:53.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 >> v1:172.21.15.100:6790/0 conn(0x7f36841457f0 legacy=0x7f36841e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:53.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 shutdown_connections 2024-03-29T11:39:53.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 >> 172.21.15.100:0/2901147174 conn(0x7f3684066a00 msgr2=0x7f36840b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:53.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 shutdown_connections 2024-03-29T11:39:53.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.121+0000 7f368c6f1640 1 -- 172.21.15.100:0/2901147174 wait complete. 2024-03-29T11:39:53.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:39:53.122+0000 7f368c6f1640 1 librados: shutdown 2024-03-29T11:39:53.137 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:59.944 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:00.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.084+0000 7f4171782640 10 monclient: get_monmap_and_config 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.084+0000 7f4171782640 10 monclient: build_initial_monmap 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.084+0000 7f4171782640 10 monclient: monmap: 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:40:00.085380+0000 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:40:00.085380+0000 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:40:00.086 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.085+0000 7f4171782640 1 Processor -- start 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.085+0000 7f4171782640 1 -- start start 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.085+0000 7f4171782640 10 monclient: init 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.085+0000 7f4171782640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.085+0000 7f4171782640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.085+0000 7f4171782640 10 monclient(hunting): picked mon.noname-c con 0x7f416c06f050 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f4171782640 10 monclient(hunting): picked mon.noname-b con 0x7f416c070240 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f41627fc640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f416c070240 0x7f416c0ae860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:42670/0 (socket says 172.21.15.100:42670) 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f41627fc640 1 -- 172.21.15.100:0/1895144337 learned_addr learned my addr 172.21.15.100:0/1895144337 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f4171782640 10 monclient(hunting): picked mon.noname-a con 0x7f416c06f430 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f4171782640 1 -- 172.21.15.100:0/1895144337 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c13f400 con 0x7f416c06f430 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f4171782640 1 -- 172.21.15.100:0/1895144337 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c1429c0 con 0x7f416c06f050 2024-03-29T11:40:00.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f4171782640 1 -- 172.21.15.100:0/1895144337 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c06ff80 con 0x7f416c070240 2024-03-29T11:40:00.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f4171782640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:00.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f4171782640 10 monclient(hunting): authenticate will time out at 1120.281368s 2024-03-29T11:40:00.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544589777 0 0) 0x7f416c06ff80 con 0x7f416c070240 2024-03-29T11:40:00.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f416a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:00.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f416a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:00.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f416a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:00.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f416a7fc640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:40:00.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.086+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4158003520 con 0x7f416c070240 2024-03-29T11:40:00.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.087+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641265973 0 0) 0x7f416c1429c0 con 0x7f416c06f050 2024-03-29T11:40:00.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.088+0000 7f416a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:00.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.088+0000 7f416a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.088+0000 7f416a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.088+0000 7f416a7fc640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.088+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f416c06ff80 con 0x7f416c06f050 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.088+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3243026277 0 0) 0x7f4158003520 con 0x7f416c070240 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.088+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 >> v1:172.21.15.191:6789/0 conn(0x7f416c06f050 legacy=0x7f416c1422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.088+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 >> v1:172.21.15.100:6789/0 conn(0x7f416c06f430 legacy=0x7f416c14a630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.089+0000 7f416a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.089+0000 7f416a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.089+0000 7f416a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.089+0000 7f416a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.089+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c0aef70 con 0x7f416c070240 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.089+0000 7f416a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:00.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4154001690 con 0x7f416c070240 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 10 monclient: dump: 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:00.094 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41540018b0 con 0x7f416c070240 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 1 -- 172.21.15.100:0/1895144337 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4154001c40 con 0x7f416c070240 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f4171782640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f4171782640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f4171782640 4 set_mon_vals no callback set 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.090+0000 7f416a7fc640 10 monclient: dump: 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:00.095 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:00.096 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:00.096 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:00.096 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:00.096 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:00.096 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:00.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.091+0000 7f4171782640 10 monclient: shutdown 2024-03-29T11:40:00.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.091+0000 7f4171782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:00.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.091+0000 7f4171782640 1 -- 172.21.15.100:0/1895144337 >> v1:172.21.15.100:6790/0 conn(0x7f416c070240 legacy=0x7f416c0ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:00.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.091+0000 7f4171782640 1 -- 172.21.15.100:0/1895144337 shutdown_connections 2024-03-29T11:40:00.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.091+0000 7f4171782640 1 -- 172.21.15.100:0/1895144337 >> 172.21.15.100:0/1895144337 conn(0x7f416c0b4930 msgr2=0x7f416c0b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:00.104 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.091+0000 7f4171782640 1 -- 172.21.15.100:0/1895144337 shutdown_connections 2024-03-29T11:40:00.104 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.091+0000 7f4171782640 1 -- 172.21.15.100:0/1895144337 wait complete. 2024-03-29T11:40:00.104 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.092+0000 7f4171782640 10 monclient: build_initial_monmap 2024-03-29T11:40:00.104 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.092+0000 7f4171782640 10 monclient: monmap: 2024-03-29T11:40:00.104 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.093+0000 7f4171782640 1 librados: starting msgr at 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.093+0000 7f4171782640 1 librados: starting objecter 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.093+0000 7f4171782640 1 Processor -- start 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.093+0000 7f4171782640 1 -- start start 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.093+0000 7f4171782640 1 librados: setting wanted keys 2024-03-29T11:40:00.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.093+0000 7f4171782640 1 librados: calling monclient init 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.093+0000 7f4171782640 10 monclient: init 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f4171782640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f4171782640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f4171782640 10 monclient(hunting): picked mon.noname-c con 0x7f416c06f050 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f416affd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f416c06f050 0x7f416c0ad6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:45206/0 (socket says 172.21.15.100:45206) 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f416affd640 1 -- 172.21.15.100:0/210800576 learned_addr learned my addr 172.21.15.100:0/210800576 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f4171782640 10 monclient(hunting): picked mon.noname-a con 0x7f416c06f430 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f4171782640 10 monclient(hunting): picked mon.noname-b con 0x7f416c070240 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0ac3b0 con 0x7f416c06f430 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0addc0 con 0x7f416c070240 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c0ac620 con 0x7f416c06f050 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f4163fff640 10 client.?.objecter ms_handle_connect 0x7f416c06f050 2024-03-29T11:40:00.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.094+0000 7f4163fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4171782640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4171782640 10 monclient(hunting): authenticate will time out at 1120.290243s 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 10 client.?.objecter ms_handle_connect 0x7f416c070240 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779734528 0 0) 0x7f416c0ac620 con 0x7f416c06f050 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f414c004eb0 con 0x7f416c06f050 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844298344 0 0) 0x7f416c0addc0 con 0x7f416c070240 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:00.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.095+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f416c0ac620 con 0x7f416c070240 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.096+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1392917079 0 0) 0x7f416c0ac620 con 0x7f416c070240 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.096+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f416c0addc0 con 0x7f416c070240 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.096+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f41640017b0 con 0x7f416c070240 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.096+0000 7f4163fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.096+0000 7f4163fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.096+0000 7f4163fff640 10 monclient(hunting): dump: 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:00.108 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.096+0000 7f4163fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.096+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3889169491 0 0) 0x7f416c0addc0 con 0x7f416c070240 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.096+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 >> v1:172.21.15.100:6790/0 conn(0x7f416c06f050 legacy=0x7f416c0ad6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.096+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 >> v1:172.21.15.100:6789/0 conn(0x7f416c06f430 legacy=0x7f416c139a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 1 monclient: found mon.b 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c0ac900 con 0x7f416c070240 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 10 monclient: _finish_auth 0 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:00.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4171782640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4171782640 10 monclient: _renew_subs 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4171782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c0ad190 con 0x7f416c070240 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4164001cc0 con 0x7f416c070240 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4164003790 con 0x7f416c070240 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4171782640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4163fff640 10 monclient: dump: 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:00.110 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4171782640 10 monclient: _renew_subs 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4171782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c0acd20 con 0x7f416c070240 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4171782640 1 librados: init done 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.098+0000 7f4169ffb640 4 set_mon_vals no callback set 2024-03-29T11:40:00.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.099+0000 7f4171782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.099+0000 7f4171782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.099+0000 7f4171782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.099+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c075f10 con 0x7f416c070240 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.101+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f4164019760 con 0x7f416c070240 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.101+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f4164083700 con 0x7f416c070240 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.101+0000 7f416affd640 10 monclient: get_auth_request con 0x7f414c06dd90 auth_method 0 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.101+0000 7f4163fff640 10 client.4351.objecter ms_dispatch 0x7f416c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.101+0000 7f4163fff640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.101+0000 7f4163fff640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.102+0000 7f4163fff640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-29T11:40:00.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.102+0000 7f4163fff640 10 client.4351.objecter ms_handle_connect 0x7f414c06dd90 2024-03-29T11:40:00.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.108+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f416404efe0 con 0x7f416c070240 2024-03-29T11:40:00.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.108+0000 7f4163fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:00.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.108+0000 7f4163fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:00.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.421+0000 7f4171782640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:00.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.421+0000 7f4171782640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:00.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.421+0000 7f4171782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:40:00.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.421+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f416c074f50 con 0x7f416c070240 2024-03-29T11:40:00.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.422+0000 7f4163fff640 1 -- 172.21.15.100:0/210800576 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f4164050f10 con 0x7f416c070240 2024-03-29T11:40:00.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.422+0000 7f4163fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:00.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.422+0000 7f4163fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:00.424 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:40:00.424 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:00.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.427+0000 7f4171782640 10 librados: watch_flush enter 2024-03-29T11:40:00.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.427+0000 7f4171782640 10 librados: watch_flush exit 2024-03-29T11:40:00.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.427+0000 7f4171782640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-29T11:40:00.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.427+0000 7f4171782640 10 client.4351.objecter successfully canceled tick 2024-03-29T11:40:00.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.427+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 >> v1:172.21.15.100:6816/3220289657 conn(0x7f414c06dd90 legacy=0x7f414c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:00.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.427+0000 7f4171782640 10 monclient: shutdown 2024-03-29T11:40:00.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.427+0000 7f4171782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:00.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.427+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 >> v1:172.21.15.191:6789/0 conn(0x7f416c070240 legacy=0x7f416c1e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:00.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.428+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 shutdown_connections 2024-03-29T11:40:00.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.428+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 >> 172.21.15.100:0/210800576 conn(0x7f416c0b4930 msgr2=0x7f416c06b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:00.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.428+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 shutdown_connections 2024-03-29T11:40:00.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.428+0000 7f4171782640 1 -- 172.21.15.100:0/210800576 wait complete. 2024-03-29T11:40:00.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:00.428+0000 7f4171782640 1 librados: shutdown 2024-03-29T11:40:00.443 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:07.249 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:07.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.399+0000 7fd8b3fff640 10 monclient: get_monmap_and_config 2024-03-29T11:40:07.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.399+0000 7fd8b3fff640 10 monclient: build_initial_monmap 2024-03-29T11:40:07.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.399+0000 7fd8b3fff640 10 monclient: monmap: 2024-03-29T11:40:07.401 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:07.401 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:07.402 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:40:07.400619+0000 2024-03-29T11:40:07.402 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:40:07.400619+0000 2024-03-29T11:40:07.402 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:07.402 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:07.402 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:07.402 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:40:07.402 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:40:07.402 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.400+0000 7fd8b3fff640 1 Processor -- start 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.400+0000 7fd8b3fff640 1 -- start start 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.400+0000 7fd8b3fff640 10 monclient: init 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.401+0000 7fd8b3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.401+0000 7fd8b3fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.401+0000 7fd8b3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd8b40b3f10 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.401+0000 7fd8b3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd8b4146440 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.401+0000 7fd8b3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd8b414fba0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.401+0000 7fd8b3fff640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b406ef50 con 0x7fd8b40b3f10 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.401+0000 7fd8b3fff640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b406f1c0 con 0x7fd8b4146440 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.401+0000 7fd8b3fff640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b4071ed0 con 0x7fd8b414fba0 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.401+0000 7fd8b3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:07.403 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.401+0000 7fd8b3fff640 10 monclient(hunting): authenticate will time out at 1127.596592s 2024-03-29T11:40:07.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.404+0000 7fd8b37fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fd8b414fba0 0x7fd8b4151f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:39488/0 (socket says 172.21.15.100:39488) 2024-03-29T11:40:07.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.404+0000 7fd8b37fe640 1 -- 172.21.15.100:0/2100119855 learned_addr learned my addr 172.21.15.100:0/2100119855 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:07.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.405+0000 7fd8b1ffb640 1 -- 172.21.15.100:0/2100119855 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207695718 0 0) 0x7fd8b4071ed0 con 0x7fd8b414fba0 2024-03-29T11:40:07.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.405+0000 7fd8b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:07.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.405+0000 7fd8b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:07.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.405+0000 7fd8b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:07.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.405+0000 7fd8b1ffb640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:40:07.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.405+0000 7fd8b1ffb640 1 -- 172.21.15.100:0/2100119855 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd89c003520 con 0x7fd8b414fba0 2024-03-29T11:40:07.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.405+0000 7fd8b1ffb640 1 -- 172.21.15.100:0/2100119855 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3403557606 0 0) 0x7fd89c003520 con 0x7fd8b414fba0 2024-03-29T11:40:07.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.405+0000 7fd8b1ffb640 1 -- 172.21.15.100:0/2100119855 >> v1:172.21.15.191:6789/0 conn(0x7fd8b4146440 legacy=0x7fd8b41468c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:07.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.405+0000 7fd8b1ffb640 1 -- 172.21.15.100:0/2100119855 >> v1:172.21.15.100:6789/0 conn(0x7fd8b40b3f10 legacy=0x7fd8b4143500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:07.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.406+0000 7fd8b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:07.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.406+0000 7fd8b1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:40:07.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.406+0000 7fd8b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:07.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.406+0000 7fd8b1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:40:07.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.406+0000 7fd8b1ffb640 1 -- 172.21.15.100:0/2100119855 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b4143c10 con 0x7fd8b414fba0 2024-03-29T11:40:07.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:07.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:07.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:07.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 1 -- 172.21.15.100:0/2100119855 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd8ac001a10 con 0x7fd8b414fba0 2024-03-29T11:40:07.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 10 monclient: dump: 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 1 -- 172.21.15.100:0/2100119855 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8ac001ca0 con 0x7fd8b414fba0 2024-03-29T11:40:07.411 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 1 -- 172.21.15.100:0/2100119855 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd8ac002030 con 0x7fd8b414fba0 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b1ffb640 10 monclient: dump: 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:07.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b3fff640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:40:07.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b3fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:07.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.407+0000 7fd8b3fff640 4 set_mon_vals no callback set 2024-03-29T11:40:07.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.408+0000 7fd8b3fff640 10 monclient: shutdown 2024-03-29T11:40:07.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.408+0000 7fd8b3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.408+0000 7fd8b3fff640 1 -- 172.21.15.100:0/2100119855 >> v1:172.21.15.100:6790/0 conn(0x7fd8b414fba0 legacy=0x7fd8b4151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.409+0000 7fd8b3fff640 1 -- 172.21.15.100:0/2100119855 shutdown_connections 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.409+0000 7fd8b3fff640 1 -- 172.21.15.100:0/2100119855 >> 172.21.15.100:0/2100119855 conn(0x7fd8b406d0c0 msgr2=0x7fd8b4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.409+0000 7fd8b3fff640 1 -- 172.21.15.100:0/2100119855 shutdown_connections 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.409+0000 7fd8b3fff640 1 -- 172.21.15.100:0/2100119855 wait complete. 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.410+0000 7fd8b3fff640 10 monclient: build_initial_monmap 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.410+0000 7fd8b3fff640 10 monclient: monmap: 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:40:07.415 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.411+0000 7fd8b3fff640 1 librados: starting msgr at 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.411+0000 7fd8b3fff640 1 librados: starting objecter 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.411+0000 7fd8b3fff640 1 Processor -- start 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.411+0000 7fd8b3fff640 1 -- start start 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.411+0000 7fd8b3fff640 1 librados: setting wanted keys 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.411+0000 7fd8b3fff640 1 librados: calling monclient init 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.411+0000 7fd8b3fff640 10 monclient: init 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.412+0000 7fd8b3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.412+0000 7fd8b3fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.412+0000 7fd8b3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd8b40b3f10 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:07.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.412+0000 7fd8b3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd8b4146440 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:07.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.413+0000 7fd8b2ffd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fd8b40b3f10 0x7fd8b414ade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:41884/0 (socket says 172.21.15.100:41884) 2024-03-29T11:40:07.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.413+0000 7fd8b3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd8b414fba0 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:07.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.413+0000 7fd8b3fff640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b414a640 con 0x7fd8b414fba0 2024-03-29T11:40:07.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.413+0000 7fd8b3fff640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b414c800 con 0x7fd8b4146440 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.413+0000 7fd8b3fff640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b414ca70 con 0x7fd8b40b3f10 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.413+0000 7fd8b3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd8b3fff640 10 monclient(hunting): authenticate will time out at 1127.609078s 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd8b2ffd640 1 -- 172.21.15.100:0/1121750521 learned_addr learned my addr 172.21.15.100:0/1121750521 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 10 client.?.objecter ms_handle_connect 0x7fd8b40b3f10 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289024565 0 0) 0x7fd8b414ca70 con 0x7fd8b40b3f10 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:40:07.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a0004eb0 con 0x7fd8b40b3f10 2024-03-29T11:40:07.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345186310 0 0) 0x7fd8a0004eb0 con 0x7fd8b40b3f10 2024-03-29T11:40:07.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8b414ca70 con 0x7fd8b40b3f10 2024-03-29T11:40:07.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd8a8001960 con 0x7fd8b40b3f10 2024-03-29T11:40:07.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:07.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3186599358 0 0) 0x7fd8b414ca70 con 0x7fd8b40b3f10 2024-03-29T11:40:07.422 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.414+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 >> v1:172.21.15.191:6789/0 conn(0x7fd8b4146440 legacy=0x7fd8b414d4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.417+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 >> v1:172.21.15.100:6789/0 conn(0x7fd8b414fba0 legacy=0x7fd8b414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.417+0000 7fd89b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.417+0000 7fd89b7fe640 1 monclient: found mon.c 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.417+0000 7fd89b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.417+0000 7fd89b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.417+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b414cd50 con 0x7fd8b40b3f10 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.417+0000 7fd89b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd89b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd89b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd8b3fff640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd8b3fff640 10 monclient: _renew_subs 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd8b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd8b3fff640 1 -- 172.21.15.100:0/1121750521 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b414b4f0 con 0x7fd8b40b3f10 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd8b3fff640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd8b3fff640 10 monclient: _renew_subs 2024-03-29T11:40:07.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd8b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd8b3fff640 1 -- 172.21.15.100:0/1121750521 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b41ef860 con 0x7fd8b40b3f10 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.418+0000 7fd8b3fff640 1 librados: init done 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.419+0000 7fd89b7fe640 10 client.4379.objecter ms_handle_connect 0x7fd8b414fba0 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.419+0000 7fd89b7fe640 10 client.4379.objecter resend_mon_ops 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.419+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8a8001c80 con 0x7fd8b40b3f10 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.419+0000 7fd89b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.419+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd8a8003a40 con 0x7fd8b40b3f10 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.419+0000 7fd89b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.419+0000 7fd89b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.419+0000 7fd89b7fe640 10 monclient: dump: 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:07.424 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:07.425 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:07.425 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:07.425 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:07.425 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:07.425 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:07.425 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:07.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.421+0000 7fd8b17fa640 4 set_mon_vals no callback set 2024-03-29T11:40:07.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.421+0000 7fd8997fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.421+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fd8a8003ba0 con 0x7fd8b40b3f10 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.422+0000 7fd8997fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.422+0000 7fd8997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.422+0000 7fd8997fa640 1 -- 172.21.15.100:0/1121750521 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b40b5ac0 con 0x7fd8b40b3f10 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.422+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fd8a8084a70 con 0x7fd8b40b3f10 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.423+0000 7fd8b27fc640 10 monclient: get_auth_request con 0x7fd8a006db70 auth_method 0 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.424+0000 7fd89b7fe640 10 client.4379.objecter ms_dispatch 0x7fd8b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.424+0000 7fd89b7fe640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.424+0000 7fd89b7fe640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.424+0000 7fd89b7fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T11:40:07.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.424+0000 7fd89b7fe640 10 client.4379.objecter ms_handle_connect 0x7fd8a006db70 2024-03-29T11:40:07.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.433+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8a80503b0 con 0x7fd8b40b3f10 2024-03-29T11:40:07.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.433+0000 7fd89b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:07.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.433+0000 7fd89b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:07.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.760+0000 7fd8997fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:07.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.760+0000 7fd8997fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:07.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.760+0000 7fd8997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:07.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.760+0000 7fd8997fa640 1 -- 172.21.15.100:0/1121750521 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8b404f730 con 0x7fd8b40b3f10 2024-03-29T11:40:07.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.762+0000 7fd89b7fe640 1 -- 172.21.15.100:0/1121750521 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fd8a80522e0 con 0x7fd8b40b3f10 2024-03-29T11:40:07.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.762+0000 7fd89b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:07.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.762+0000 7fd89b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:07.764 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:40:07.764 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:07.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.766+0000 7fd8997fa640 10 librados: watch_flush enter 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.766+0000 7fd8997fa640 10 librados: watch_flush exit 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.766+0000 7fd8997fa640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.766+0000 7fd8997fa640 10 client.4379.objecter successfully canceled tick 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.767+0000 7fd8997fa640 1 -- 172.21.15.100:0/1121750521 >> v1:172.21.15.100:6816/3220289657 conn(0x7fd8a006db70 legacy=0x7fd8a0070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.767+0000 7fd8997fa640 10 monclient: shutdown 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.767+0000 7fd8997fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.767+0000 7fd8997fa640 1 -- 172.21.15.100:0/1121750521 >> v1:172.21.15.100:6790/0 conn(0x7fd8b40b3f10 legacy=0x7fd8b414ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.767+0000 7fd8997fa640 1 -- 172.21.15.100:0/1121750521 shutdown_connections 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.767+0000 7fd8997fa640 1 -- 172.21.15.100:0/1121750521 >> 172.21.15.100:0/1121750521 conn(0x7fd8b406d0c0 msgr2=0x7fd8b40b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.767+0000 7fd8997fa640 1 -- 172.21.15.100:0/1121750521 shutdown_connections 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.767+0000 7fd8997fa640 1 -- 172.21.15.100:0/1121750521 wait complete. 2024-03-29T11:40:07.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:07.767+0000 7fd8997fa640 1 librados: shutdown 2024-03-29T11:40:07.785 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:14.592 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:14.736 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.733+0000 7f4ca440f640 10 monclient: get_monmap_and_config 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.733+0000 7f4ca440f640 10 monclient: build_initial_monmap 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.733+0000 7f4ca440f640 10 monclient: monmap: 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:40:14.734596+0000 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:40:14.734596+0000 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:40:14.737 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:40:14.738 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:14.738 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.735+0000 7f4ca440f640 1 Processor -- start 2024-03-29T11:40:14.740 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.735+0000 7f4ca440f640 1 -- start start 2024-03-29T11:40:14.740 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.735+0000 7f4ca440f640 10 monclient: init 2024-03-29T11:40:14.740 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca440f640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:14.740 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca440f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:14.740 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca440f640 10 monclient(hunting): picked mon.noname-c con 0x7f4c9c0b0770 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:14.740 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca440f640 10 monclient(hunting): picked mon.noname-b con 0x7f4c9c14a700 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:14.740 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca440f640 10 monclient(hunting): picked mon.noname-a con 0x7f4c9c0ac5a0 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca440f640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9c0aba70 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca440f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9c0aecb0 con 0x7f4c9c0b0770 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca440f640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9c0af150 con 0x7f4c9c14a700 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca440f640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca440f640 10 monclient(hunting): authenticate will time out at 1134.931672s 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca19aa640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f4c9c14a700 0x7f4c9c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:43872/0 (socket says 172.21.15.100:43872) 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.736+0000 7f4ca19aa640 1 -- 172.21.15.100:0/726309069 learned_addr learned my addr 172.21.15.100:0/726309069 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.738+0000 7f4ca11a9640 1 -- 172.21.15.100:0/726309069 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511539020 0 0) 0x7f4c9c0af150 con 0x7f4c9c14a700 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.738+0000 7f4ca11a9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.738+0000 7f4ca11a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.738+0000 7f4ca11a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.738+0000 7f4ca11a9640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.738+0000 7f4ca11a9640 1 -- 172.21.15.100:0/726309069 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c88003520 con 0x7f4c9c14a700 2024-03-29T11:40:14.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.738+0000 7f4ca11a9640 1 -- 172.21.15.100:0/726309069 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2474848498 0 0) 0x7f4c88003520 con 0x7f4c9c14a700 2024-03-29T11:40:14.742 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.738+0000 7f4ca11a9640 1 -- 172.21.15.100:0/726309069 >> v1:172.21.15.191:6789/0 conn(0x7f4c9c0b0770 legacy=0x7f4c9c149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 1 -- 172.21.15.100:0/726309069 >> v1:172.21.15.100:6789/0 conn(0x7f4c9c0ac5a0 legacy=0x7f4c9c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 1 monclient: found mon.noname-b 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 1 -- 172.21.15.100:0/726309069 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c0ac9d0 con 0x7f4c9c14a700 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 10 monclient: _finish_auth 0 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca440f640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca440f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:14.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 1 -- 172.21.15.100:0/726309069 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4c98001690 con 0x7f4c9c14a700 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca11a9640 10 monclient: dump: 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:14.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.740+0000 7f4ca440f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.742+0000 7f4ca11a9640 1 -- 172.21.15.100:0/726309069 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c98001920 con 0x7f4c9c14a700 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.742+0000 7f4ca11a9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.742+0000 7f4ca440f640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.742+0000 7f4ca440f640 4 set_mon_vals no callback set 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.742+0000 7f4ca11a9640 1 -- 172.21.15.100:0/726309069 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4c98001cb0 con 0x7f4c9c14a700 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.742+0000 7f4ca11a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.742+0000 7f4ca11a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.742+0000 7f4ca11a9640 10 monclient: dump: 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:14.746 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.743+0000 7f4ca440f640 10 monclient: shutdown 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.743+0000 7f4ca440f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.743+0000 7f4ca440f640 1 -- 172.21.15.100:0/726309069 >> v1:172.21.15.100:6790/0 conn(0x7f4c9c14a700 legacy=0x7f4c9c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.743+0000 7f4ca440f640 1 -- 172.21.15.100:0/726309069 shutdown_connections 2024-03-29T11:40:14.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.743+0000 7f4ca440f640 1 -- 172.21.15.100:0/726309069 >> 172.21.15.100:0/726309069 conn(0x7f4c9c06b660 msgr2=0x7f4c9c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:14.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.745+0000 7f4ca440f640 1 -- 172.21.15.100:0/726309069 shutdown_connections 2024-03-29T11:40:14.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.745+0000 7f4ca440f640 1 -- 172.21.15.100:0/726309069 wait complete. 2024-03-29T11:40:14.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.745+0000 7f4ca440f640 10 monclient: build_initial_monmap 2024-03-29T11:40:14.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.745+0000 7f4ca440f640 10 monclient: monmap: 2024-03-29T11:40:14.748 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:14.748 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:14.748 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:40:14.748 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:40:14.748 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:14.748 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.746+0000 7f4ca440f640 1 librados: starting msgr at 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.746+0000 7f4ca440f640 1 librados: starting objecter 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.746+0000 7f4ca440f640 1 Processor -- start 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.746+0000 7f4ca440f640 1 -- start start 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.746+0000 7f4ca440f640 1 librados: setting wanted keys 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.746+0000 7f4ca440f640 1 librados: calling monclient init 2024-03-29T11:40:14.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.746+0000 7f4ca440f640 10 monclient: init 2024-03-29T11:40:14.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca440f640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:14.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca440f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:14.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca440f640 10 monclient(hunting): picked mon.noname-c con 0x7f4c9c0ac5a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca440f640 10 monclient(hunting): picked mon.noname-a con 0x7f4c9c0b0770 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca440f640 10 monclient(hunting): picked mon.noname-b con 0x7f4c9c14a700 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca440f640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9c07e340 con 0x7f4c9c0b0770 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca440f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9c1eca00 con 0x7f4c9c14a700 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca440f640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9c1ed630 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca440f640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca440f640 10 monclient(hunting): authenticate will time out at 1134.943253s 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca21ab640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f4c9c0ac5a0 0x7f4c9c1ecf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:49220/0 (socket says 172.21.15.100:49220) 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4ca21ab640 1 -- 172.21.15.100:0/3400989136 learned_addr learned my addr 172.21.15.100:0/3400989136 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4c92ffd640 10 client.?.objecter ms_handle_connect 0x7f4c9c0ac5a0 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.748+0000 7f4c92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.749+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094047909 0 0) 0x7f4c9c1ed630 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.749+0000 7f4c92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:14.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.749+0000 7f4c92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:14.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.749+0000 7f4c92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:14.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.749+0000 7f4c92ffd640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:40:14.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.749+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c80004eb0 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1495425276 0 0) 0x7f4c80004eb0 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c80005300 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4c8c001830 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 10 monclient(hunting): dump: 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:14.757 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3352897572 0 0) 0x7f4c80005300 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 >> v1:172.21.15.191:6789/0 conn(0x7f4c9c14a700 legacy=0x7f4c9c1f8180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 >> v1:172.21.15.100:6789/0 conn(0x7f4c9c0b0770 legacy=0x7f4c9c1e99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 1 monclient: found mon.c 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c1ed7c0 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4c92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4ca440f640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4ca440f640 10 monclient: _renew_subs 2024-03-29T11:40:14.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4ca440f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4ca440f640 1 -- 172.21.15.100:0/3400989136 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9c07c860 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4ca440f640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4ca440f640 10 monclient: _renew_subs 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4ca440f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4ca440f640 1 -- 172.21.15.100:0/3400989136 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9c07b1d0 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.752+0000 7f4ca440f640 1 librados: init done 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.753+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c8c001e00 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.753+0000 7f4c92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.753+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4c8c001830 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.753+0000 7f4c92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.753+0000 7f4c92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.753+0000 7f4c92ffd640 10 monclient: dump: 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:14.759 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.753+0000 7f4ca09a8640 4 set_mon_vals no callback set 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.754+0000 7f4ca440f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.754+0000 7f4ca440f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.754+0000 7f4ca440f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:14.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.754+0000 7f4ca440f640 1 -- 172.21.15.100:0/3400989136 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c84005080 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.756+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f4c8c01a8b0 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.756+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f4c8c084650 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.756+0000 7f4c92ffd640 10 client.4385.objecter ms_dispatch 0x7f4c9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:14.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.756+0000 7f4c92ffd640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:14.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.756+0000 7f4c92ffd640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:14.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.756+0000 7f4c92ffd640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:40:14.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.757+0000 7f4ca19aa640 10 monclient: get_auth_request con 0x7f4c8006e240 auth_method 0 2024-03-29T11:40:14.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.757+0000 7f4c92ffd640 10 client.4385.objecter ms_handle_connect 0x7f4c8006e240 2024-03-29T11:40:14.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.764+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c8c04fd60 con 0x7f4c9c0ac5a0 2024-03-29T11:40:14.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.764+0000 7f4c92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:14.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:14.764+0000 7f4c92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:15.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.085+0000 7f4ca440f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:15.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.085+0000 7f4ca440f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:15.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.085+0000 7f4ca440f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:15.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.085+0000 7f4ca440f640 1 -- 172.21.15.100:0/3400989136 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c84005330 con 0x7f4c9c0ac5a0 2024-03-29T11:40:15.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.087+0000 7f4c92ffd640 1 -- 172.21.15.100:0/3400989136 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f4c8c001690 con 0x7f4c9c0ac5a0 2024-03-29T11:40:15.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.087+0000 7f4c92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:15.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.087+0000 7f4c92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:15.090 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:40:15.090 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:15.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.091+0000 7f4ca440f640 10 librados: watch_flush enter 2024-03-29T11:40:15.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.091+0000 7f4ca440f640 10 librados: watch_flush exit 2024-03-29T11:40:15.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.091+0000 7f4ca440f640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:40:15.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.091+0000 7f4ca440f640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:40:15.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.091+0000 7f4ca440f640 1 -- 172.21.15.100:0/3400989136 >> v1:172.21.15.100:6816/3220289657 conn(0x7f4c8006e240 legacy=0x7f4c80070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:15.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.091+0000 7f4ca440f640 10 monclient: shutdown 2024-03-29T11:40:15.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.091+0000 7f4ca440f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:15.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.091+0000 7f4ca440f640 1 -- 172.21.15.100:0/3400989136 >> v1:172.21.15.100:6790/0 conn(0x7f4c9c0ac5a0 legacy=0x7f4c9c1ecf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:15.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.092+0000 7f4ca440f640 1 -- 172.21.15.100:0/3400989136 shutdown_connections 2024-03-29T11:40:15.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.092+0000 7f4ca440f640 1 -- 172.21.15.100:0/3400989136 >> 172.21.15.100:0/3400989136 conn(0x7f4c9c06b660 msgr2=0x7f4c9c0b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:15.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.093+0000 7f4ca440f640 1 -- 172.21.15.100:0/3400989136 shutdown_connections 2024-03-29T11:40:15.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.093+0000 7f4ca440f640 1 -- 172.21.15.100:0/3400989136 wait complete. 2024-03-29T11:40:15.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:15.093+0000 7f4ca440f640 1 librados: shutdown 2024-03-29T11:40:15.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:21.920 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:22.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.070+0000 7f9e1fac8640 10 monclient: get_monmap_and_config 2024-03-29T11:40:22.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.070+0000 7f9e1fac8640 10 monclient: build_initial_monmap 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.070+0000 7f9e1fac8640 10 monclient: monmap: 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:40:22.071890+0000 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:40:22.071890+0000 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:40:22.073 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:22.074 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.072+0000 7f9e1fac8640 1 Processor -- start 2024-03-29T11:40:22.074 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.073+0000 7f9e1fac8640 1 -- start start 2024-03-29T11:40:22.075 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.073+0000 7f9e1fac8640 10 monclient: init 2024-03-29T11:40:22.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.074+0000 7f9e1fac8640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:22.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.074+0000 7f9e1fac8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:22.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.074+0000 7f9e1fac8640 10 monclient(hunting): picked mon.noname-c con 0x7f9e180b0770 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:22.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.074+0000 7f9e1fac8640 10 monclient(hunting): picked mon.noname-a con 0x7f9e1814a700 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:22.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.074+0000 7f9e1e065640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f9e180ac5a0 0x7f9e18151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:38574/0 (socket says 172.21.15.100:38574) 2024-03-29T11:40:22.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.074+0000 7f9e1e065640 1 -- 172.21.15.100:0/4236432175 learned_addr learned my addr 172.21.15.100:0/4236432175 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:22.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.074+0000 7f9e1fac8640 10 monclient(hunting): picked mon.noname-b con 0x7f9e180ac5a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:22.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.074+0000 7f9e1fac8640 1 -- 172.21.15.100:0/4236432175 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e180aba70 con 0x7f9e1814a700 2024-03-29T11:40:22.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.074+0000 7f9e1fac8640 1 -- 172.21.15.100:0/4236432175 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e180aecb0 con 0x7f9e180b0770 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.074+0000 7f9e1fac8640 1 -- 172.21.15.100:0/4236432175 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e180af150 con 0x7f9e180ac5a0 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.075+0000 7f9e1fac8640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.075+0000 7f9e1fac8640 10 monclient(hunting): authenticate will time out at 1142.270093s 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.075+0000 7f9e1c862640 1 -- 172.21.15.100:0/4236432175 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693381810 0 0) 0x7f9e180af150 con 0x7f9e180ac5a0 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.075+0000 7f9e1c862640 10 monclient(hunting): _negotiate 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.075+0000 7f9e1c862640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.075+0000 7f9e1c862640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.075+0000 7f9e1c862640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.075+0000 7f9e1c862640 1 -- 172.21.15.100:0/4236432175 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e04003520 con 0x7f9e180ac5a0 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.075+0000 7f9e1c862640 1 -- 172.21.15.100:0/4236432175 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2183879376 0 0) 0x7f9e04003520 con 0x7f9e180ac5a0 2024-03-29T11:40:22.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.075+0000 7f9e1c862640 1 -- 172.21.15.100:0/4236432175 >> v1:172.21.15.191:6789/0 conn(0x7f9e180b0770 legacy=0x7f9e18149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:22.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.077+0000 7f9e1c862640 1 -- 172.21.15.100:0/4236432175 >> v1:172.21.15.100:6789/0 conn(0x7f9e1814a700 legacy=0x7f9e180abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:22.080 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: _finish_hunting 0 2024-03-29T11:40:22.080 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 1 monclient: found mon.noname-b 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 1 -- 172.21.15.100:0/4236432175 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e180ac9d0 con 0x7f9e180ac5a0 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: _finish_auth 0 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: _check_auth_tickets 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 1 -- 172.21.15.100:0/4236432175 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f9e14001690 con 0x7f9e180ac5a0 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: dump: 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:22.081 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:22.082 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:22.082 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:22.082 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:22.082 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:22.082 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:22.082 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:22.082 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1fac8640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:40:22.082 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1fac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:22.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 1 -- 172.21.15.100:0/4236432175 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e140018f0 con 0x7f9e180ac5a0 2024-03-29T11:40:22.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:22.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 1 -- 172.21.15.100:0/4236432175 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f9e14001d40 con 0x7f9e180ac5a0 2024-03-29T11:40:22.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1fac8640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1fac8640 4 set_mon_vals no callback set 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.079+0000 7f9e1c862640 10 monclient: dump: 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:22.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.080+0000 7f9e1fac8640 10 monclient: shutdown 2024-03-29T11:40:22.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.080+0000 7f9e1fac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:22.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.080+0000 7f9e1fac8640 1 -- 172.21.15.100:0/4236432175 >> v1:172.21.15.100:6790/0 conn(0x7f9e180ac5a0 legacy=0x7f9e18151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:22.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.080+0000 7f9e1fac8640 1 -- 172.21.15.100:0/4236432175 shutdown_connections 2024-03-29T11:40:22.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.080+0000 7f9e1fac8640 1 -- 172.21.15.100:0/4236432175 >> 172.21.15.100:0/4236432175 conn(0x7f9e1806b660 msgr2=0x7f9e1806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:22.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.081+0000 7f9e1fac8640 1 -- 172.21.15.100:0/4236432175 shutdown_connections 2024-03-29T11:40:22.085 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.082+0000 7f9e1fac8640 1 -- 172.21.15.100:0/4236432175 wait complete. 2024-03-29T11:40:22.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.083+0000 7f9e1fac8640 10 monclient: build_initial_monmap 2024-03-29T11:40:22.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.083+0000 7f9e1fac8640 10 monclient: monmap: 2024-03-29T11:40:22.086 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:22.086 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:22.086 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:40:22.086 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:40:22.086 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:22.086 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.083+0000 7f9e1fac8640 1 librados: starting msgr at 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.083+0000 7f9e1fac8640 1 librados: starting objecter 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.083+0000 7f9e1fac8640 1 Processor -- start 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.083+0000 7f9e1fac8640 1 -- start start 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.083+0000 7f9e1fac8640 1 librados: setting wanted keys 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.083+0000 7f9e1fac8640 1 librados: calling monclient init 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.083+0000 7f9e1fac8640 10 monclient: init 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.084+0000 7f9e1fac8640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.084+0000 7f9e1fac8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.084+0000 7f9e1fac8640 10 monclient(hunting): picked mon.noname-c con 0x7f9e180ac5a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:22.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.084+0000 7f9e1fac8640 10 monclient(hunting): picked mon.noname-b con 0x7f9e180b0770 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.085+0000 7f9e1d864640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f9e180ac5a0 0x7f9e18126690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:39886/0 (socket says 172.21.15.100:39886) 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.085+0000 7f9e1fac8640 10 monclient(hunting): picked mon.noname-a con 0x7f9e1814a700 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.085+0000 7f9e1fac8640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e18127110 con 0x7f9e1814a700 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.085+0000 7f9e1fac8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e18125c40 con 0x7f9e180b0770 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.085+0000 7f9e1fac8640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e18126da0 con 0x7f9e180ac5a0 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.085+0000 7f9e1fac8640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.085+0000 7f9e1fac8640 10 monclient(hunting): authenticate will time out at 1142.281051s 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.086+0000 7f9e1d864640 1 -- 172.21.15.100:0/459992201 learned_addr learned my addr 172.21.15.100:0/459992201 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.086+0000 7f9e0e7fc640 10 client.?.objecter ms_handle_connect 0x7f9e180ac5a0 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.086+0000 7f9e0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.086+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809929825 0 0) 0x7f9e18126da0 con 0x7f9e180ac5a0 2024-03-29T11:40:22.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.086+0000 7f9e0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:22.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.086+0000 7f9e0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:22.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.087+0000 7f9e0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:22.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.088+0000 7f9e0e7fc640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.088+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dfc004eb0 con 0x7f9e180ac5a0 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.088+0000 7f9e0e7fc640 10 client.?.objecter ms_handle_connect 0x7f9e180b0770 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.088+0000 7f9e0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.088+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051592592 0 0) 0x7f9e18125c40 con 0x7f9e180b0770 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.089+0000 7f9e0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.089+0000 7f9e0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.089+0000 7f9e0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.089+0000 7f9e0e7fc640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.089+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e18126da0 con 0x7f9e180b0770 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.089+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 65102641 0 0) 0x7f9dfc004eb0 con 0x7f9e180ac5a0 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e18125c40 con 0x7f9e180ac5a0 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f9e08001870 con 0x7f9e180ac5a0 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:22.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:22.093 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 10 client.?.objecter ms_handle_connect 0x7f9e1814a700 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3234951141 0 0) 0x7f9e18126da0 con 0x7f9e180b0770 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dfc004eb0 con 0x7f9e180b0770 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f9e10001b10 con 0x7f9e180b0770 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.090+0000 7f9e0e7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:22.095 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.091+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4163974518 0 0) 0x7f9e18125c40 con 0x7f9e180ac5a0 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.091+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 >> v1:172.21.15.191:6789/0 conn(0x7f9e180b0770 legacy=0x7f9e18144d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 >> v1:172.21.15.100:6789/0 conn(0x7f9e1814a700 legacy=0x7f9e181db600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e0e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e0e7fc640 1 monclient: found mon.c 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e0e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:22.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18145a50 con 0x7f9e180ac5a0 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e0e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e0e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e1fac8640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e1fac8640 10 monclient: _renew_subs 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e1fac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e1fac8640 1 -- 172.21.15.100:0/459992201 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e18145470 con 0x7f9e180ac5a0 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e1fac8640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e1fac8640 10 monclient: _renew_subs 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e1fac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e1fac8640 1 -- 172.21.15.100:0/459992201 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e181dbd10 con 0x7f9e180ac5a0 2024-03-29T11:40:22.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.092+0000 7f9e1fac8640 1 librados: init done 2024-03-29T11:40:22.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.095+0000 7f9e1fac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:22.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.095+0000 7f9e1fac8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:22.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.095+0000 7f9e1fac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:22.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.095+0000 7f9e1fac8640 1 -- 172.21.15.100:0/459992201 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e180ab560 con 0x7f9e180ac5a0 2024-03-29T11:40:22.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.096+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e080019d0 con 0x7f9e180ac5a0 2024-03-29T11:40:22.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.096+0000 7f9e0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:22.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.096+0000 7f9e0ffff640 4 set_mon_vals no callback set 2024-03-29T11:40:22.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.097+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f9e08004aa0 con 0x7f9e180ac5a0 2024-03-29T11:40:22.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.097+0000 7f9e0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:22.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.097+0000 7f9e0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.097+0000 7f9e0e7fc640 10 monclient: dump: 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:22.100 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:22.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.098+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f9e08004c40 con 0x7f9e180ac5a0 2024-03-29T11:40:22.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.099+0000 7f9e1d063640 10 monclient: get_auth_request con 0x7f9dfc06dde0 auth_method 0 2024-03-29T11:40:22.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.099+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f9e080849f0 con 0x7f9e180ac5a0 2024-03-29T11:40:22.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.099+0000 7f9e0e7fc640 10 client.4391.objecter ms_dispatch 0x7f9e18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:22.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.099+0000 7f9e0e7fc640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:22.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.099+0000 7f9e0e7fc640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:22.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.099+0000 7f9e0e7fc640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-29T11:40:22.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.099+0000 7f9e0e7fc640 10 client.4391.objecter ms_handle_connect 0x7f9dfc06dde0 2024-03-29T11:40:22.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.105+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9e080502d0 con 0x7f9e180ac5a0 2024-03-29T11:40:22.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.105+0000 7f9e0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:22.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.105+0000 7f9e0e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:22.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.432+0000 7f9e1fac8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:22.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.432+0000 7f9e1fac8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:22.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.432+0000 7f9e1fac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:22.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.432+0000 7f9e1fac8640 1 -- 172.21.15.100:0/459992201 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e18126f30 con 0x7f9e180ac5a0 2024-03-29T11:40:22.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.435+0000 7f9e0e7fc640 1 -- 172.21.15.100:0/459992201 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f9e08052200 con 0x7f9e180ac5a0 2024-03-29T11:40:22.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.435+0000 7f9e0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:22.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.435+0000 7f9e0e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:22.437 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:40:22.437 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:22.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.438+0000 7f9e1fac8640 10 librados: watch_flush enter 2024-03-29T11:40:22.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.439+0000 7f9e1fac8640 10 librados: watch_flush exit 2024-03-29T11:40:22.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.439+0000 7f9e1fac8640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-29T11:40:22.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.439+0000 7f9e1fac8640 10 client.4391.objecter successfully canceled tick 2024-03-29T11:40:22.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.439+0000 7f9e1fac8640 1 -- 172.21.15.100:0/459992201 >> v1:172.21.15.100:6816/3220289657 conn(0x7f9dfc06dde0 legacy=0x7f9dfc0702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:22.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.439+0000 7f9e1fac8640 10 monclient: shutdown 2024-03-29T11:40:22.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.440+0000 7f9e1fac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:22.442 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.440+0000 7f9e1fac8640 1 -- 172.21.15.100:0/459992201 >> v1:172.21.15.100:6790/0 conn(0x7f9e180ac5a0 legacy=0x7f9e18126690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:22.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.443+0000 7f9e1fac8640 1 -- 172.21.15.100:0/459992201 shutdown_connections 2024-03-29T11:40:22.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.443+0000 7f9e1fac8640 1 -- 172.21.15.100:0/459992201 >> 172.21.15.100:0/459992201 conn(0x7f9e1806b660 msgr2=0x7f9e180b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:22.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.443+0000 7f9e1fac8640 1 -- 172.21.15.100:0/459992201 shutdown_connections 2024-03-29T11:40:22.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.443+0000 7f9e1fac8640 1 -- 172.21.15.100:0/459992201 wait complete. 2024-03-29T11:40:22.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:22.443+0000 7f9e1fac8640 1 librados: shutdown 2024-03-29T11:40:22.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:29.271 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:29.425 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.422+0000 7f508d5ed640 10 monclient: get_monmap_and_config 2024-03-29T11:40:29.425 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.422+0000 7f508d5ed640 10 monclient: build_initial_monmap 2024-03-29T11:40:29.425 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.422+0000 7f508d5ed640 10 monclient: monmap: 2024-03-29T11:40:29.425 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:29.425 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:29.425 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:40:29.423172+0000 2024-03-29T11:40:29.425 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:40:29.423172+0000 2024-03-29T11:40:29.425 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:29.426 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:29.426 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:29.426 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:40:29.426 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:40:29.426 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:29.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.423+0000 7f508d5ed640 1 Processor -- start 2024-03-29T11:40:29.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.423+0000 7f508d5ed640 1 -- start start 2024-03-29T11:40:29.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.423+0000 7f508d5ed640 10 monclient: init 2024-03-29T11:40:29.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.424+0000 7f508d5ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:29.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.424+0000 7f508d5ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:29.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.424+0000 7f508d5ed640 10 monclient(hunting): picked mon.noname-b con 0x7f50880a96c0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:29.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.424+0000 7f508d5ed640 10 monclient(hunting): picked mon.noname-a con 0x7f50880a9aa0 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:29.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.424+0000 7f508d5ed640 10 monclient(hunting): picked mon.noname-c con 0x7f50880ad730 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:29.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.424+0000 7f508d5ed640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508813fee0 con 0x7f50880a9aa0 2024-03-29T11:40:29.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.424+0000 7f508d5ed640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50880aab80 con 0x7f50880ad730 2024-03-29T11:40:29.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.424+0000 7f508d5ed640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5088145e80 con 0x7f50880a96c0 2024-03-29T11:40:29.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.424+0000 7f508d5ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:29.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.424+0000 7f508d5ed640 10 monclient(hunting): authenticate will time out at 1149.619233s 2024-03-29T11:40:29.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.428+0000 7f50877fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f50880ad730 0x7f508814dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:57953/0 (socket says 172.21.15.100:57953) 2024-03-29T11:40:29.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.428+0000 7f50877fe640 1 -- 172.21.15.100:0/1681694946 learned_addr learned my addr 172.21.15.100:0/1681694946 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:29.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.430+0000 7f5085ffb640 1 -- 172.21.15.100:0/1681694946 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260159523 0 0) 0x7f50880aab80 con 0x7f50880ad730 2024-03-29T11:40:29.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.430+0000 7f5085ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:29.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.430+0000 7f5085ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:29.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.430+0000 7f5085ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:29.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.430+0000 7f5085ffb640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:40:29.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.430+0000 7f5085ffb640 1 -- 172.21.15.100:0/1681694946 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5070003520 con 0x7f50880ad730 2024-03-29T11:40:29.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 1 -- 172.21.15.100:0/1681694946 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 362113150 0 0) 0x7f5070003520 con 0x7f50880ad730 2024-03-29T11:40:29.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 1 -- 172.21.15.100:0/1681694946 >> v1:172.21.15.100:6790/0 conn(0x7f50880a96c0 legacy=0x7f508813f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:29.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 1 -- 172.21.15.100:0/1681694946 >> v1:172.21.15.100:6789/0 conn(0x7f50880a9aa0 legacy=0x7f5088145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:29.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:29.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 1 monclient: found mon.noname-c 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 1 -- 172.21.15.100:0/1681694946 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50880b4440 con 0x7f50880ad730 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 1 -- 172.21.15.100:0/1681694946 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f507c001690 con 0x7f50880ad730 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.431+0000 7f5085ffb640 10 monclient: dump: 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.434 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.435 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:29.435 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:29.435 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:29.435 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:29.435 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:29.435 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:29.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.432+0000 7f5085ffb640 1 -- 172.21.15.100:0/1681694946 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f507c001860 con 0x7f50880ad730 2024-03-29T11:40:29.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.432+0000 7f5085ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:29.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.432+0000 7f5085ffb640 1 -- 172.21.15.100:0/1681694946 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f507c001cb0 con 0x7f50880ad730 2024-03-29T11:40:29.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.432+0000 7f5085ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:29.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.432+0000 7f5085ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:29.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.432+0000 7f5085ffb640 10 monclient: dump: 2024-03-29T11:40:29.436 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.432+0000 7f508d5ed640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.432+0000 7f508d5ed640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.432+0000 7f508d5ed640 4 set_mon_vals no callback set 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.433+0000 7f508d5ed640 10 monclient: shutdown 2024-03-29T11:40:29.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.433+0000 7f508d5ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:29.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.433+0000 7f508d5ed640 1 -- 172.21.15.100:0/1681694946 >> v1:172.21.15.191:6789/0 conn(0x7f50880ad730 legacy=0x7f508814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:29.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.435+0000 7f508d5ed640 1 -- 172.21.15.100:0/1681694946 shutdown_connections 2024-03-29T11:40:29.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.435+0000 7f508d5ed640 1 -- 172.21.15.100:0/1681694946 >> 172.21.15.100:0/1681694946 conn(0x7f50880b35e0 msgr2=0x7f50880b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:29.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.436+0000 7f508d5ed640 1 -- 172.21.15.100:0/1681694946 shutdown_connections 2024-03-29T11:40:29.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.436+0000 7f508d5ed640 1 -- 172.21.15.100:0/1681694946 wait complete. 2024-03-29T11:40:29.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.437+0000 7f508d5ed640 10 monclient: build_initial_monmap 2024-03-29T11:40:29.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.437+0000 7f508d5ed640 10 monclient: monmap: 2024-03-29T11:40:29.439 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:29.439 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:29.439 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:40:29.439 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:40:29.440 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:29.440 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:29.440 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:29.440 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:40:29.440 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:40:29.440 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:29.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.438+0000 7f508d5ed640 1 librados: starting msgr at 2024-03-29T11:40:29.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.438+0000 7f508d5ed640 1 librados: starting objecter 2024-03-29T11:40:29.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.439+0000 7f508d5ed640 1 Processor -- start 2024-03-29T11:40:29.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.439+0000 7f508d5ed640 1 -- start start 2024-03-29T11:40:29.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.439+0000 7f508d5ed640 1 librados: setting wanted keys 2024-03-29T11:40:29.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.439+0000 7f508d5ed640 1 librados: calling monclient init 2024-03-29T11:40:29.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.439+0000 7f508d5ed640 10 monclient: init 2024-03-29T11:40:29.442 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.440+0000 7f508d5ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:29.442 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.440+0000 7f508d5ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:29.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.442+0000 7f508d5ed640 10 monclient(hunting): picked mon.noname-c con 0x7f50880a96c0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:29.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.442+0000 7f508d5ed640 10 monclient(hunting): picked mon.noname-a con 0x7f50880ad730 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:29.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.442+0000 7f508d5ed640 10 monclient(hunting): picked mon.noname-b con 0x7f508813c0f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:29.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.442+0000 7f508d5ed640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508813e4e0 con 0x7f50880ad730 2024-03-29T11:40:29.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.442+0000 7f508d5ed640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508813e7c0 con 0x7f508813c0f0 2024-03-29T11:40:29.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.442+0000 7f508d5ed640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508813c5d0 con 0x7f50880a96c0 2024-03-29T11:40:29.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.442+0000 7f508d5ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:29.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.442+0000 7f508d5ed640 10 monclient(hunting): authenticate will time out at 1149.637270s 2024-03-29T11:40:29.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.445+0000 7f5086ffd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f50880a96c0 0x7f5088137c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:41089/0 (socket says 172.21.15.100:41089) 2024-03-29T11:40:29.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.445+0000 7f5086ffd640 1 -- 172.21.15.100:0/2088335669 learned_addr learned my addr 172.21.15.100:0/2088335669 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:29.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.447+0000 7f506f7fe640 10 client.?.objecter ms_handle_connect 0x7f50880ad730 2024-03-29T11:40:29.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.447+0000 7f506f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.447+0000 7f506f7fe640 10 client.?.objecter ms_handle_connect 0x7f50880a96c0 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.447+0000 7f506f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 client.?.objecter ms_handle_connect 0x7f508813c0f0 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861580505 0 0) 0x7f508813e4e0 con 0x7f50880ad730 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5074004eb0 con 0x7f50880ad730 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198061430 0 0) 0x7f508813c5d0 con 0x7f50880a96c0 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:29.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f508813e4e0 con 0x7f50880a96c0 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911458719 0 0) 0x7f508813e7c0 con 0x7f508813c0f0 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f508813c5d0 con 0x7f508813c0f0 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1072929839 0 0) 0x7f508813e4e0 con 0x7f50880a96c0 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f508813e7c0 con 0x7f50880a96c0 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5078001750 con 0x7f50880a96c0 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:29.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1101899210 0 0) 0x7f5074004eb0 con 0x7f50880ad730 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f508813e4e0 con 0x7f50880ad730 2024-03-29T11:40:29.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5080001bf0 con 0x7f50880ad730 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4215588336 0 0) 0x7f508813c5d0 con 0x7f508813c0f0 2024-03-29T11:40:29.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5074004eb0 con 0x7f508813c0f0 2024-03-29T11:40:29.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f507c0019d0 con 0x7f508813c0f0 2024-03-29T11:40:29.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:29.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:29.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.448+0000 7f506f7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:29.460 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:29.460 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:29.461 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.461 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.461 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:29.461 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:29.461 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:29.461 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:29.461 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2126248565 0 0) 0x7f508813e4e0 con 0x7f50880ad730 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 >> v1:172.21.15.100:6790/0 conn(0x7f50880a96c0 legacy=0x7f5088137c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 >> v1:172.21.15.191:6789/0 conn(0x7f508813c0f0 legacy=0x7f50881d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 1 monclient: found mon.a 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508813d520 con 0x7f50880ad730 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:29.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f506f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f508d5ed640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f508d5ed640 10 monclient: _renew_subs 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f508d5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f508d5ed640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508813cb00 con 0x7f50880ad730 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f508d5ed640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f508d5ed640 10 monclient: _renew_subs 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f508d5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f508d5ed640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5088138390 con 0x7f50880ad730 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.449+0000 7f508d5ed640 1 librados: init done 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.450+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5080003970 con 0x7f50880ad730 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.450+0000 7f506f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.450+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5080002100 con 0x7f50880ad730 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.450+0000 7f506f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:29.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.450+0000 7f506f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.450+0000 7f506f7fe640 10 monclient: dump: 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.450+0000 7f508d5ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.450+0000 7f508d5ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.450+0000 7f508d5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:40:29.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.450+0000 7f508d5ed640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5088075f10 con 0x7f50880ad730 2024-03-29T11:40:29.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.451+0000 7f50857fa640 4 set_mon_vals no callback set 2024-03-29T11:40:29.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.452+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f5080003c80 con 0x7f50880ad730 2024-03-29T11:40:29.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.452+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f5080083a80 con 0x7f50880ad730 2024-03-29T11:40:29.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.452+0000 7f506f7fe640 10 client.4359.objecter ms_dispatch 0x7f5088000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:29.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.452+0000 7f506f7fe640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:29.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.452+0000 7f506f7fe640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:29.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.452+0000 7f506f7fe640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-29T11:40:29.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.453+0000 7f5086ffd640 10 monclient: get_auth_request con 0x7f5074076970 auth_method 0 2024-03-29T11:40:29.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.454+0000 7f506f7fe640 10 client.4359.objecter ms_handle_connect 0x7f5074076970 2024-03-29T11:40:29.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.459+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f508004f190 con 0x7f50880ad730 2024-03-29T11:40:29.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.459+0000 7f506f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:29.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.459+0000 7f506f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:29.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.786+0000 7f508d5ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:29.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.786+0000 7f508d5ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:29.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.786+0000 7f508d5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:40:29.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.786+0000 7f508d5ed640 1 -- 172.21.15.100:0/2088335669 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50880b24c0 con 0x7f50880ad730 2024-03-29T11:40:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.791+0000 7f506f7fe640 1 -- 172.21.15.100:0/2088335669 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f5080051290 con 0x7f50880ad730 2024-03-29T11:40:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.791+0000 7f506f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.791+0000 7f506f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:29.796 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:40:29.796 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:29.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.797+0000 7f508d5ed640 10 librados: watch_flush enter 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.797+0000 7f508d5ed640 10 librados: watch_flush exit 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.797+0000 7f508d5ed640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.797+0000 7f508d5ed640 10 client.4359.objecter successfully canceled tick 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.798+0000 7f508d5ed640 1 -- 172.21.15.100:0/2088335669 >> v1:172.21.15.100:6816/3220289657 conn(0x7f5074076970 legacy=0x7f5074078e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.798+0000 7f508d5ed640 10 monclient: shutdown 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.798+0000 7f508d5ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.798+0000 7f508d5ed640 1 -- 172.21.15.100:0/2088335669 >> v1:172.21.15.100:6789/0 conn(0x7f50880ad730 legacy=0x7f50881d7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.798+0000 7f508d5ed640 1 -- 172.21.15.100:0/2088335669 shutdown_connections 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.798+0000 7f508d5ed640 1 -- 172.21.15.100:0/2088335669 >> 172.21.15.100:0/2088335669 conn(0x7f50880b35e0 msgr2=0x7f508813f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.798+0000 7f508d5ed640 1 -- 172.21.15.100:0/2088335669 shutdown_connections 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.798+0000 7f508d5ed640 1 -- 172.21.15.100:0/2088335669 wait complete. 2024-03-29T11:40:29.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:29.798+0000 7f508d5ed640 1 librados: shutdown 2024-03-29T11:40:29.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:36.626 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:36.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.785+0000 7f8c97fff640 10 monclient: get_monmap_and_config 2024-03-29T11:40:36.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.785+0000 7f8c97fff640 10 monclient: build_initial_monmap 2024-03-29T11:40:36.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.786+0000 7f8c97fff640 10 monclient: monmap: 2024-03-29T11:40:36.788 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:36.788 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:36.788 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:40:36.787012+0000 2024-03-29T11:40:36.788 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:40:36.787012+0000 2024-03-29T11:40:36.788 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:36.789 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:36.789 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:36.789 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:40:36.789 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:40:36.789 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:36.790 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.787+0000 7f8c97fff640 1 Processor -- start 2024-03-29T11:40:36.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.787+0000 7f8c97fff640 1 -- start start 2024-03-29T11:40:36.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.787+0000 7f8c97fff640 10 monclient: init 2024-03-29T11:40:36.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.788+0000 7f8c97fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:36.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.788+0000 7f8c97fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:36.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.788+0000 7f8c97fff640 10 monclient(hunting): picked mon.noname-a con 0x7f8c90006a70 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:36.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.788+0000 7f8c97fff640 10 monclient(hunting): picked mon.noname-c con 0x7f8c9002b100 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:36.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.788+0000 7f8c97fff640 10 monclient(hunting): picked mon.noname-b con 0x7f8c90029c30 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:36.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.788+0000 7f8c97fff640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9002a270 con 0x7f8c90006a70 2024-03-29T11:40:36.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.788+0000 7f8c97fff640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9002c440 con 0x7f8c9002b100 2024-03-29T11:40:36.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.788+0000 7f8c97fff640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c900c9440 con 0x7f8c90029c30 2024-03-29T11:40:36.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.788+0000 7f8c97fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:36.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.788+0000 7f8c97fff640 10 monclient(hunting): authenticate will time out at 1156.983368s 2024-03-29T11:40:36.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.791+0000 7f8c967fc640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f8c9002b100 0x7f8c900cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:49709/0 (socket says 172.21.15.100:49709) 2024-03-29T11:40:36.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.792+0000 7f8c967fc640 1 -- 172.21.15.100:0/1310353048 learned_addr learned my addr 172.21.15.100:0/1310353048 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:36.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.796+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813067461 0 0) 0x7f8c9002a270 con 0x7f8c90006a70 2024-03-29T11:40:36.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.796+0000 7f8c95ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:36.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.796+0000 7f8c95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:36.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.796+0000 7f8c95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:36.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.796+0000 7f8c95ffb640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:40:36.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.796+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c80003520 con 0x7f8c90006a70 2024-03-29T11:40:36.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872160552 0 0) 0x7f8c9002c440 con 0x7f8c9002b100 2024-03-29T11:40:36.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c95ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c95ffb640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9002a270 con 0x7f8c9002b100 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c977fe640 1 -- 172.21.15.100:0/1310353048 >> v1:172.21.15.100:6790/0 conn(0x7f8c90029c30 legacy=0x7f8c900d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c977fe640 1 -- 172.21.15.100:0/1310353048 >> v1:172.21.15.100:6790/0 conn(0x7f8c90029c30 legacy=0x7f8c900d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3145834196 0 0) 0x7f8c80003520 con 0x7f8c90006a70 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 >> v1:172.21.15.100:6790/0 conn(0x7f8c90029c30 legacy=0x7f8c900d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.797+0000 7f8c977fe640 1 --1- 172.21.15.100:0/1310353048 >> v1:172.21.15.100:6790/0 conn(0x7f8c90029c30 0x7f8c900d74a0 :-1 s=OPENED pgs=394494 cs=1 l=1).handle_message read tag failed 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c977fe640 1 --1- 172.21.15.100:0/1310353048 >> v1:172.21.15.100:6790/0 conn(0x7f8c90029c30 0x7f8c900d74a0 :-1 s=OPENED pgs=394494 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 >> v1:172.21.15.191:6789/0 conn(0x7f8c9002b100 legacy=0x7f8c900cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 1 monclient: found mon.noname-a 2024-03-29T11:40:36.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c900c95d0 con 0x7f8c90006a70 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: ms_handle_reset stray mon v1:172.21.15.100:6790/0 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8c9804f730 con 0x7f8c90006a70 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: dump: 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:36.841 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c97fff640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c97fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c98075f10 con 0x7f8c90006a70 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 1 -- 172.21.15.100:0/1310353048 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8c980758b0 con 0x7f8c90006a70 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.798+0000 7f8c95ffb640 10 monclient: dump: 2024-03-29T11:40:36.842 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.799+0000 7f8c97fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.799+0000 7f8c97fff640 4 set_mon_vals no callback set 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.799+0000 7f8c97fff640 10 monclient: shutdown 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.799+0000 7f8c97fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:36.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.799+0000 7f8c97fff640 1 -- 172.21.15.100:0/1310353048 >> v1:172.21.15.100:6789/0 conn(0x7f8c90006a70 legacy=0x7f8c900c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.800+0000 7f8c97fff640 1 -- 172.21.15.100:0/1310353048 shutdown_connections 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.800+0000 7f8c97fff640 1 -- 172.21.15.100:0/1310353048 >> 172.21.15.100:0/1310353048 conn(0x7f8c900058f0 msgr2=0x7f8c90005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.805+0000 7f8c97fff640 1 -- 172.21.15.100:0/1310353048 shutdown_connections 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.805+0000 7f8c97fff640 1 -- 172.21.15.100:0/1310353048 wait complete. 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.806+0000 7f8c97fff640 10 monclient: build_initial_monmap 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.806+0000 7f8c97fff640 10 monclient: monmap: 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:36.844 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.807+0000 7f8c97fff640 1 librados: starting msgr at 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.807+0000 7f8c97fff640 1 librados: starting objecter 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.807+0000 7f8c97fff640 1 Processor -- start 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.807+0000 7f8c97fff640 1 -- start start 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.807+0000 7f8c97fff640 1 librados: setting wanted keys 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.807+0000 7f8c97fff640 1 librados: calling monclient init 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.807+0000 7f8c97fff640 10 monclient: init 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.808+0000 7f8c97fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.808+0000 7f8c97fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.808+0000 7f8c97fff640 10 monclient(hunting): picked mon.noname-c con 0x7f8c90006a70 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.808+0000 7f8c97fff640 10 monclient(hunting): picked mon.noname-b con 0x7f8c90029c30 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:36.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.812+0000 7f8c97fff640 10 monclient(hunting): picked mon.noname-a con 0x7f8c9002da20 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.812+0000 7f8c97fff640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c901695d0 con 0x7f8c9002da20 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.812+0000 7f8c97fff640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9016a170 con 0x7f8c90029c30 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.812+0000 7f8c97fff640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c9002d270 con 0x7f8c90006a70 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.812+0000 7f8c97fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.812+0000 7f8c97fff640 10 monclient(hunting): authenticate will time out at 1157.007725s 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.813+0000 7f8c967fc640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f8c90029c30 0x7f8c9002cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:49835/0 (socket says 172.21.15.100:49835) 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.813+0000 7f8c967fc640 1 -- 172.21.15.100:0/1624502396 learned_addr learned my addr 172.21.15.100:0/1624502396 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.815+0000 7f8c7f7fe640 10 client.?.objecter ms_handle_connect 0x7f8c90029c30 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.815+0000 7f8c7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.815+0000 7f8c7f7fe640 10 client.?.objecter ms_handle_connect 0x7f8c9002da20 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.815+0000 7f8c7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126044709 0 0) 0x7f8c9016a170 con 0x7f8c90029c30 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:36.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c84004eb0 con 0x7f8c90029c30 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199057430 0 0) 0x7f8c901695d0 con 0x7f8c9002da20 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.816+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9016a170 con 0x7f8c9002da20 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1759532924 0 0) 0x7f8c84004eb0 con 0x7f8c90029c30 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c901695d0 con 0x7f8c90029c30 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2603485952 0 0) 0x7f8c9016a170 con 0x7f8c9002da20 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c84004eb0 con 0x7f8c9002da20 2024-03-29T11:40:36.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8c88001890 con 0x7f8c90029c30 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:36.848 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8c8c0017d0 con 0x7f8c9002da20 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:36.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3800481047 0 0) 0x7f8c901695d0 con 0x7f8c90029c30 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 >> v1:172.21.15.100:6790/0 conn(0x7f8c90006a70 legacy=0x7f8c90169a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 >> v1:172.21.15.100:6789/0 conn(0x7f8c9002da20 legacy=0x7f8c901664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 monclient: found mon.b 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c900ccb70 con 0x7f8c90029c30 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c97fff640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c97fff640 10 monclient: _renew_subs 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c97fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:40:36.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c97fff640 1 -- 172.21.15.100:0/1624502396 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c90166be0 con 0x7f8c90029c30 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c97fff640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c97fff640 10 monclient: _renew_subs 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.817+0000 7f8c97fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.818+0000 7f8c97fff640 1 -- 172.21.15.100:0/1624502396 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c900cacb0 con 0x7f8c90029c30 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.818+0000 7f8c97fff640 1 librados: init done 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.818+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c880019f0 con 0x7f8c90029c30 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.818+0000 7f8c7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.818+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8c88004a00 con 0x7f8c90029c30 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.818+0000 7f8c7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.818+0000 7f8c7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.818+0000 7f8c7f7fe640 10 monclient: dump: 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:36.851 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.818+0000 7f8c957fa640 4 set_mon_vals no callback set 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.820+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f8c8801a9d0 con 0x7f8c90029c30 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.820+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f8c88084a30 con 0x7f8c90029c30 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.820+0000 7f8c7f7fe640 10 client.4366.objecter ms_dispatch 0x7f8c98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.820+0000 7f8c7f7fe640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.821+0000 7f8c7f7fe640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.821+0000 7f8c97fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.821+0000 7f8c7f7fe640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T11:40:36.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.821+0000 7f8c97fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:36.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.821+0000 7f8c97fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:40:36.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.821+0000 7f8c97fff640 1 -- 172.21.15.100:0/1624502396 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c78005080 con 0x7f8c90029c30 2024-03-29T11:40:36.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.822+0000 7f8c96ffd640 10 monclient: get_auth_request con 0x7f8c8406df10 auth_method 0 2024-03-29T11:40:36.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.823+0000 7f8c7f7fe640 10 client.4366.objecter ms_handle_connect 0x7f8c8406df10 2024-03-29T11:40:36.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.829+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c88050140 con 0x7f8c90029c30 2024-03-29T11:40:36.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.830+0000 7f8c7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:36.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:36.830+0000 7f8c7f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:37.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.211+0000 7f8c97fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:37.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.211+0000 7f8c97fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:37.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.211+0000 7f8c97fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:40:37.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.211+0000 7f8c97fff640 1 -- 172.21.15.100:0/1624502396 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c78005330 con 0x7f8c90029c30 2024-03-29T11:40:37.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.212+0000 7f8c7f7fe640 1 -- 172.21.15.100:0/1624502396 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f8c88052240 con 0x7f8c90029c30 2024-03-29T11:40:37.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.212+0000 7f8c7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:37.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.212+0000 7f8c7f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:37.215 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:40:37.215 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:37.217 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 10 librados: watch_flush enter 2024-03-29T11:40:37.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 10 librados: watch_flush exit 2024-03-29T11:40:37.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T11:40:37.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 10 client.4366.objecter successfully canceled tick 2024-03-29T11:40:37.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 1 -- 172.21.15.100:0/1624502396 >> v1:172.21.15.100:6816/3220289657 conn(0x7f8c8406df10 legacy=0x7f8c840703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:37.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 10 monclient: shutdown 2024-03-29T11:40:37.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:37.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 1 -- 172.21.15.100:0/1624502396 >> v1:172.21.15.191:6789/0 conn(0x7f8c90029c30 legacy=0x7f8c9002cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:37.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 1 -- 172.21.15.100:0/1624502396 shutdown_connections 2024-03-29T11:40:37.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 1 -- 172.21.15.100:0/1624502396 >> 172.21.15.100:0/1624502396 conn(0x7f8c900058f0 msgr2=0x7f8c900c7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:37.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 1 -- 172.21.15.100:0/1624502396 shutdown_connections 2024-03-29T11:40:37.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.216+0000 7f8c97fff640 1 -- 172.21.15.100:0/1624502396 wait complete. 2024-03-29T11:40:37.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:37.217+0000 7f8c97fff640 1 librados: shutdown 2024-03-29T11:40:37.233 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:44.040 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:44.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.194+0000 7f748c9ec640 10 monclient: get_monmap_and_config 2024-03-29T11:40:44.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.194+0000 7f748c9ec640 10 monclient: build_initial_monmap 2024-03-29T11:40:44.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.194+0000 7f748c9ec640 10 monclient: monmap: 2024-03-29T11:40:44.196 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:44.196 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:44.196 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:40:44.195403+0000 2024-03-29T11:40:44.197 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:40:44.195403+0000 2024-03-29T11:40:44.197 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:44.197 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:44.197 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:44.197 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:40:44.197 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:40:44.197 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:44.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.195+0000 7f748c9ec640 1 Processor -- start 2024-03-29T11:40:44.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.195+0000 7f748c9ec640 1 -- start start 2024-03-29T11:40:44.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.195+0000 7f748c9ec640 10 monclient: init 2024-03-29T11:40:44.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.196+0000 7f748c9ec640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:44.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.196+0000 7f748c9ec640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:44.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.196+0000 7f748c9ec640 10 monclient(hunting): picked mon.noname-c con 0x7f74880b4820 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:44.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.196+0000 7f748c9ec640 10 monclient(hunting): picked mon.noname-a con 0x7f74880a9ce0 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:44.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.196+0000 7f748c9ec640 10 monclient(hunting): picked mon.noname-b con 0x7f74880b3cd0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:44.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.196+0000 7f748c9ec640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74880b4180 con 0x7f74880a9ce0 2024-03-29T11:40:44.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.196+0000 7f748c9ec640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74880aaf60 con 0x7f74880b4820 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.196+0000 7f748c9ec640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748813c200 con 0x7f74880b3cd0 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.196+0000 7f748c9ec640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.196+0000 7f748c9ec640 10 monclient(hunting): authenticate will time out at 1164.391691s 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.197+0000 7f7487fff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f74880b3cd0 0x7f748814d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:42786/0 (socket says 172.21.15.100:42786) 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.197+0000 7f7487fff640 1 -- 172.21.15.100:0/1937048724 learned_addr learned my addr 172.21.15.100:0/1937048724 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.198+0000 7f74867fc640 1 -- 172.21.15.100:0/1937048724 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377563770 0 0) 0x7f748813c200 con 0x7f74880b3cd0 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.198+0000 7f74867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.198+0000 7f74867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.198+0000 7f74867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.198+0000 7f74867fc640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:40:44.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.198+0000 7f74867fc640 1 -- 172.21.15.100:0/1937048724 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7470003520 con 0x7f74880b3cd0 2024-03-29T11:40:44.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.199+0000 7f74867fc640 1 -- 172.21.15.100:0/1937048724 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3355922011 0 0) 0x7f7470003520 con 0x7f74880b3cd0 2024-03-29T11:40:44.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.199+0000 7f74867fc640 1 -- 172.21.15.100:0/1937048724 >> v1:172.21.15.191:6789/0 conn(0x7f74880b4820 legacy=0x7f748813baf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:44.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.199+0000 7f74867fc640 1 -- 172.21.15.100:0/1937048724 >> v1:172.21.15.100:6789/0 conn(0x7f74880a9ce0 legacy=0x7f7488145770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 1 monclient: found mon.noname-b 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 1 -- 172.21.15.100:0/1937048724 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7488145e80 con 0x7f74880b3cd0 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 1 -- 172.21.15.100:0/1937048724 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f7480001a10 con 0x7f74880b3cd0 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:44.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.201+0000 7f74867fc640 10 monclient: dump: 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.202+0000 7f748c9ec640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:40:44.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.202+0000 7f748c9ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:44.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.205+0000 7f74867fc640 1 -- 172.21.15.100:0/1937048724 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7480001be0 con 0x7f74880b3cd0 2024-03-29T11:40:44.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.205+0000 7f74867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:44.208 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.205+0000 7f748c9ec640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:44.208 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.205+0000 7f748c9ec640 4 set_mon_vals no callback set 2024-03-29T11:40:44.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.206+0000 7f748c9ec640 10 monclient: shutdown 2024-03-29T11:40:44.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.206+0000 7f748c9ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:44.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.206+0000 7f748c9ec640 1 -- 172.21.15.100:0/1937048724 >> v1:172.21.15.100:6790/0 conn(0x7f74880b3cd0 legacy=0x7f748814d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:44.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.207+0000 7f748c9ec640 1 -- 172.21.15.100:0/1937048724 shutdown_connections 2024-03-29T11:40:44.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.207+0000 7f748c9ec640 1 -- 172.21.15.100:0/1937048724 >> 172.21.15.100:0/1937048724 conn(0x7f748806d070 msgr2=0x7f7488066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:44.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.209+0000 7f748c9ec640 1 -- 172.21.15.100:0/1937048724 shutdown_connections 2024-03-29T11:40:44.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.209+0000 7f748c9ec640 1 -- 172.21.15.100:0/1937048724 wait complete. 2024-03-29T11:40:44.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.210+0000 7f748c9ec640 10 monclient: build_initial_monmap 2024-03-29T11:40:44.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.210+0000 7f748c9ec640 10 monclient: monmap: 2024-03-29T11:40:44.213 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:44.213 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:44.213 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:40:44.213 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:40:44.213 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:44.213 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:44.213 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:44.213 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:40:44.214 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:40:44.214 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:44.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.211+0000 7f748c9ec640 1 librados: starting msgr at 2024-03-29T11:40:44.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.211+0000 7f748c9ec640 1 librados: starting objecter 2024-03-29T11:40:44.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.211+0000 7f748c9ec640 1 Processor -- start 2024-03-29T11:40:44.215 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.211+0000 7f748c9ec640 1 -- start start 2024-03-29T11:40:44.215 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.211+0000 7f748c9ec640 1 librados: setting wanted keys 2024-03-29T11:40:44.215 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.211+0000 7f748c9ec640 1 librados: calling monclient init 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.211+0000 7f748c9ec640 10 monclient: init 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.212+0000 7f748c9ec640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.212+0000 7f748c9ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.212+0000 7f748c9ec640 10 monclient(hunting): picked mon.noname-a con 0x7f74880a9ce0 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.212+0000 7f748c9ec640 10 monclient(hunting): picked mon.noname-c con 0x7f74880b3cd0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f748c9ec640 10 monclient(hunting): picked mon.noname-b con 0x7f74880b4820 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f748c9ec640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748813e0a0 con 0x7f74880a9ce0 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f748c9ec640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748813d800 con 0x7f74880b4820 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f748c9ec640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748813b480 con 0x7f74880b3cd0 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f748c9ec640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f748c9ec640 10 monclient(hunting): authenticate will time out at 1164.408256s 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f7486ffd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f74880b3cd0 0x7f74881d71e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:48490/0 (socket says 172.21.15.100:48490) 2024-03-29T11:40:44.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f7486ffd640 1 -- 172.21.15.100:0/4018839909 learned_addr learned my addr 172.21.15.100:0/4018839909 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:44.217 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f746ffff640 10 client.?.objecter ms_handle_connect 0x7f74880b3cd0 2024-03-29T11:40:44.217 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f746ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:44.217 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310726299 0 0) 0x7f748813b480 con 0x7f74880b3cd0 2024-03-29T11:40:44.217 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f746ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:44.217 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f746ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:44.217 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f746ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:44.217 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f746ffff640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:40:44.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.213+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7474004eb0 con 0x7f74880b3cd0 2024-03-29T11:40:44.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.214+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 96875609 0 0) 0x7f7474004eb0 con 0x7f74880b3cd0 2024-03-29T11:40:44.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.214+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f748813b480 con 0x7f74880b3cd0 2024-03-29T11:40:44.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.214+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f747c001960 con 0x7f74880b3cd0 2024-03-29T11:40:44.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.214+0000 7f746ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:44.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.214+0000 7f746ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:44.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.214+0000 7f746ffff640 10 monclient(hunting): dump: 2024-03-29T11:40:44.218 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.214+0000 7f746ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.214+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2368051861 0 0) 0x7f748813b480 con 0x7f74880b3cd0 2024-03-29T11:40:44.219 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.214+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 >> v1:172.21.15.191:6789/0 conn(0x7f74880b4820 legacy=0x7f74881d78f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:44.221 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.216+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 >> v1:172.21.15.100:6789/0 conn(0x7f74880a9ce0 legacy=0x7f748813e5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:44.221 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f746ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:44.221 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f746ffff640 1 monclient: found mon.c 2024-03-29T11:40:44.221 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f746ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f746ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748813b610 con 0x7f74880b3cd0 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f746ffff640 10 monclient: _finish_auth 0 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f746ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f746ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f748c9ec640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f748c9ec640 10 monclient: _renew_subs 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f748c9ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.217+0000 7f748c9ec640 1 -- 172.21.15.100:0/4018839909 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748813ecb0 con 0x7f74880b3cd0 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.218+0000 7f748c9ec640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.218+0000 7f748c9ec640 10 monclient: _renew_subs 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.218+0000 7f748c9ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:44.222 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.218+0000 7f748c9ec640 1 -- 172.21.15.100:0/4018839909 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74881e2c50 con 0x7f74880b3cd0 2024-03-29T11:40:44.223 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.218+0000 7f748c9ec640 1 librados: init done 2024-03-29T11:40:44.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.219+0000 7f746dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:44.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.219+0000 7f746dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:44.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.219+0000 7f746dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:44.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.219+0000 7f746dffb640 1 -- 172.21.15.100:0/4018839909 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7468005080 con 0x7f74880b3cd0 2024-03-29T11:40:44.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.220+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f747c001ac0 con 0x7f74880b3cd0 2024-03-29T11:40:44.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.220+0000 7f746ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:44.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.220+0000 7f7485ffb640 4 set_mon_vals no callback set 2024-03-29T11:40:44.224 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.221+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f747c0038a0 con 0x7f74880b3cd0 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.221+0000 7f746ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.222+0000 7f746ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.222+0000 7f746ffff640 10 monclient: dump: 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:44.225 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.222+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f747c003ac0 con 0x7f74880b3cd0 2024-03-29T11:40:44.226 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.223+0000 7f74877fe640 10 monclient: get_auth_request con 0x7f747406dc80 auth_method 0 2024-03-29T11:40:44.227 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.223+0000 7f746ffff640 10 client.4400.objecter ms_handle_connect 0x7f747406dc80 2024-03-29T11:40:44.227 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.223+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f747c084a50 con 0x7f74880b3cd0 2024-03-29T11:40:44.227 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.223+0000 7f746ffff640 10 client.4400.objecter ms_dispatch 0x7f7488000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:44.227 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.223+0000 7f746ffff640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:44.227 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.223+0000 7f746ffff640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:44.227 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.223+0000 7f746ffff640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-29T11:40:44.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.229+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f747c0503b0 con 0x7f74880b3cd0 2024-03-29T11:40:44.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.229+0000 7f746ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:44.231 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.229+0000 7f746ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:44.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.554+0000 7f746dffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:44.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.554+0000 7f746dffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:44.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.554+0000 7f746dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:44.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.554+0000 7f746dffb640 1 -- 172.21.15.100:0/4018839909 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7468005330 con 0x7f74880b3cd0 2024-03-29T11:40:44.560 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.558+0000 7f746ffff640 1 -- 172.21.15.100:0/4018839909 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f747c0522e0 con 0x7f74880b3cd0 2024-03-29T11:40:44.560 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.558+0000 7f746ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:44.560 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.558+0000 7f746ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:44.560 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:40:44.560 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:44.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.561+0000 7f746dffb640 10 librados: watch_flush enter 2024-03-29T11:40:44.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.561+0000 7f746dffb640 10 librados: watch_flush exit 2024-03-29T11:40:44.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.561+0000 7f746dffb640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-29T11:40:44.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.561+0000 7f746dffb640 10 client.4400.objecter successfully canceled tick 2024-03-29T11:40:44.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.562+0000 7f746dffb640 1 -- 172.21.15.100:0/4018839909 >> v1:172.21.15.100:6816/3220289657 conn(0x7f747406dc80 legacy=0x7f7474070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:44.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.562+0000 7f746dffb640 10 monclient: shutdown 2024-03-29T11:40:44.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.562+0000 7f746dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:44.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.562+0000 7f746dffb640 1 -- 172.21.15.100:0/4018839909 >> v1:172.21.15.100:6790/0 conn(0x7f74880b3cd0 legacy=0x7f74881d71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:44.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.562+0000 7f746dffb640 1 -- 172.21.15.100:0/4018839909 shutdown_connections 2024-03-29T11:40:44.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.562+0000 7f746dffb640 1 -- 172.21.15.100:0/4018839909 >> 172.21.15.100:0/4018839909 conn(0x7f748806d070 msgr2=0x7f74880b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:44.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.562+0000 7f746dffb640 1 -- 172.21.15.100:0/4018839909 shutdown_connections 2024-03-29T11:40:44.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.562+0000 7f746dffb640 1 -- 172.21.15.100:0/4018839909 wait complete. 2024-03-29T11:40:44.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:44.563+0000 7f746dffb640 1 librados: shutdown 2024-03-29T11:40:44.586 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:51.393 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:51.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.539+0000 7f86a19dc640 10 monclient: get_monmap_and_config 2024-03-29T11:40:51.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.539+0000 7f86a19dc640 10 monclient: build_initial_monmap 2024-03-29T11:40:51.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.539+0000 7f86a19dc640 10 monclient: monmap: 2024-03-29T11:40:51.541 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:51.541 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:51.541 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:40:51.540133+0000 2024-03-29T11:40:51.541 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:40:51.540133+0000 2024-03-29T11:40:51.542 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:51.542 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:51.542 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:51.542 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:40:51.542 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:40:51.542 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:51.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.540+0000 7f86a19dc640 1 Processor -- start 2024-03-29T11:40:51.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.540+0000 7f86a19dc640 1 -- start start 2024-03-29T11:40:51.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.540+0000 7f86a19dc640 10 monclient: init 2024-03-29T11:40:51.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.541+0000 7f86a19dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:51.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.541+0000 7f86a19dc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:51.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.541+0000 7f86a19dc640 10 monclient(hunting): picked mon.noname-b con 0x7f86940066e0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:51.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.541+0000 7f86a19dc640 10 monclient(hunting): picked mon.noname-c con 0x7f86940cf820 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:51.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.541+0000 7f86a19dc640 10 monclient(hunting): picked mon.noname-a con 0x7f86940c92b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:51.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.541+0000 7f86a19dc640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869402c100 con 0x7f86940c92b0 2024-03-29T11:40:51.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.541+0000 7f86a19dc640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869402c3e0 con 0x7f86940cf820 2024-03-29T11:40:51.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.541+0000 7f86a19dc640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869402a030 con 0x7f86940066e0 2024-03-29T11:40:51.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.541+0000 7f86a19dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:51.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.541+0000 7f86a19dc640 10 monclient(hunting): authenticate will time out at 1171.736498s 2024-03-29T11:40:51.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.542+0000 7f86a11db640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f86940c92b0 0x7f86940d74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:55410/0 (socket says 172.21.15.100:55410) 2024-03-29T11:40:51.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.542+0000 7f86a11db640 1 -- 172.21.15.100:0/4276104753 learned_addr learned my addr 172.21.15.100:0/4276104753 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:51.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.542+0000 7f869b7fe640 1 -- 172.21.15.100:0/4276104753 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547607312 0 0) 0x7f869402c100 con 0x7f86940c92b0 2024-03-29T11:40:51.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.542+0000 7f869b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:51.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.542+0000 7f869b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:51.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.542+0000 7f869b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:51.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.542+0000 7f869b7fe640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:40:51.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.542+0000 7f869b7fe640 1 -- 172.21.15.100:0/4276104753 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868c003520 con 0x7f86940c92b0 2024-03-29T11:40:51.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 1 -- 172.21.15.100:0/4276104753 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288701303 0 0) 0x7f868c003520 con 0x7f86940c92b0 2024-03-29T11:40:51.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 1 -- 172.21.15.100:0/4276104753 >> v1:172.21.15.100:6790/0 conn(0x7f86940066e0 legacy=0x7f86940cf110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:51.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 1 -- 172.21.15.100:0/4276104753 >> v1:172.21.15.191:6789/0 conn(0x7f86940cf820 legacy=0x7f86940c8ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:51.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:51.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:40:51.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:51.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:40:51.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 1 -- 172.21.15.100:0/4276104753 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8694029ce0 con 0x7f86940c92b0 2024-03-29T11:40:51.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 1 -- 172.21.15.100:0/4276104753 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8690001690 con 0x7f86940c92b0 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f869b7fe640 10 monclient: dump: 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:51.547 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:51.548 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:51.548 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:51.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f86a19dc640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:40:51.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.543+0000 7f86a19dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.544+0000 7f869b7fe640 1 -- 172.21.15.100:0/4276104753 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8690001860 con 0x7f86940c92b0 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.544+0000 7f869b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.544+0000 7f86a19dc640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.544+0000 7f86a19dc640 4 set_mon_vals no callback set 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.545+0000 7f86a19dc640 10 monclient: shutdown 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.545+0000 7f86a19dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.545+0000 7f86a19dc640 1 -- 172.21.15.100:0/4276104753 >> v1:172.21.15.100:6789/0 conn(0x7f86940c92b0 legacy=0x7f86940d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.545+0000 7f86a19dc640 1 -- 172.21.15.100:0/4276104753 shutdown_connections 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.545+0000 7f86a19dc640 1 -- 172.21.15.100:0/4276104753 >> 172.21.15.100:0/4276104753 conn(0x7f86940058f0 msgr2=0x7f8694005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.545+0000 7f86a19dc640 1 -- 172.21.15.100:0/4276104753 shutdown_connections 2024-03-29T11:40:51.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.545+0000 7f86a19dc640 1 -- 172.21.15.100:0/4276104753 wait complete. 2024-03-29T11:40:51.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.546+0000 7f86a19dc640 10 monclient: build_initial_monmap 2024-03-29T11:40:51.554 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.546+0000 7f86a19dc640 10 monclient: monmap: 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.547+0000 7f86a19dc640 1 librados: starting msgr at 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.547+0000 7f86a19dc640 1 librados: starting objecter 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.547+0000 7f86a19dc640 1 Processor -- start 2024-03-29T11:40:51.555 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.547+0000 7f86a19dc640 1 -- start start 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.547+0000 7f86a19dc640 1 librados: setting wanted keys 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.547+0000 7f86a19dc640 1 librados: calling monclient init 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.547+0000 7f86a19dc640 10 monclient: init 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86a19dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86a19dc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86a19dc640 10 monclient(hunting): picked mon.noname-c con 0x7f86940066e0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86a19dc640 10 monclient(hunting): picked mon.noname-a con 0x7f86940c92b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86a19dc640 10 monclient(hunting): picked mon.noname-b con 0x7f86940cf820 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86a19dc640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86940cc4f0 con 0x7f86940c92b0 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86a19dc640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869402d2b0 con 0x7f86940cf820 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86a19dc640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86940cc980 con 0x7f86940066e0 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86a19dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86a19dc640 10 monclient(hunting): authenticate will time out at 1171.743685s 2024-03-29T11:40:51.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f869bfff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f86940c92b0 0x7f8694031e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:55520/0 (socket says 172.21.15.100:55520) 2024-03-29T11:40:51.557 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f869bfff640 1 -- 172.21.15.100:0/2749673749 learned_addr learned my addr 172.21.15.100:0/2749673749 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:51.557 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86997fa640 10 client.?.objecter ms_handle_connect 0x7f86940c92b0 2024-03-29T11:40:51.557 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.548+0000 7f86997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:51.557 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.549+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424624905 0 0) 0x7f86940cc4f0 con 0x7f86940c92b0 2024-03-29T11:40:51.557 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.549+0000 7f86997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:40:51.557 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.549+0000 7f86997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:51.557 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.549+0000 7f86997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:51.557 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.549+0000 7f86997fa640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:40:51.557 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.549+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8680004eb0 con 0x7f86940c92b0 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657619112 0 0) 0x7f8680004eb0 con 0x7f86940c92b0 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86940cc4f0 con 0x7f86940c92b0 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f869c04fdd0 con 0x7f86940c92b0 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 10 monclient(hunting): dump: 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:51.558 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:51.559 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:51.559 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:51.559 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 10 client.?.objecter ms_handle_connect 0x7f86940066e0 2024-03-29T11:40:51.559 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:51.559 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3497958361 0 0) 0x7f86940cc4f0 con 0x7f86940c92b0 2024-03-29T11:40:51.561 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 >> v1:172.21.15.100:6790/0 conn(0x7f86940066e0 legacy=0x7f8694182b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:51.561 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.550+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 >> v1:172.21.15.191:6789/0 conn(0x7f86940cf820 legacy=0x7f869418e4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:51.561 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86997fa640 1 monclient: found mon.a 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86940ccb10 con 0x7f86940c92b0 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86997fa640 10 monclient: _finish_auth 0 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86a19dc640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86a19dc640 10 monclient: _renew_subs 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86a19dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86a19dc640 1 -- 172.21.15.100:0/2749673749 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86940cba80 con 0x7f86940c92b0 2024-03-29T11:40:51.562 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86a19dc640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86a19dc640 10 monclient: _renew_subs 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86a19dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86a19dc640 1 -- 172.21.15.100:0/2749673749 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8694183470 con 0x7f86940c92b0 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.553+0000 7f86a19dc640 1 librados: init done 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.555+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f869c04f400 con 0x7f86940c92b0 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.555+0000 7f86997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.555+0000 7f869a7fc640 4 set_mon_vals no callback set 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.555+0000 7f8686ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.556+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f869c075710 con 0x7f86940c92b0 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.556+0000 7f86997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.556+0000 7f86997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.556+0000 7f86997fa640 10 monclient: dump: 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:51.563 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.556+0000 7f8686ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.556+0000 7f8686ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.556+0000 7f8686ffd640 1 -- 172.21.15.100:0/2749673749 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867c005080 con 0x7f86940c92b0 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.558+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f869c067bf0 con 0x7f86940c92b0 2024-03-29T11:40:51.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.559+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f869c0ee560 con 0x7f86940c92b0 2024-03-29T11:40:51.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.559+0000 7f86997fa640 10 client.4371.objecter ms_dispatch 0x7f869c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:51.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.559+0000 7f86997fa640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:51.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.559+0000 7f86997fa640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:51.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.559+0000 7f86997fa640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-29T11:40:51.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.560+0000 7f86a09da640 10 monclient: get_auth_request con 0x7f868006daa0 auth_method 0 2024-03-29T11:40:51.565 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.560+0000 7f86997fa640 10 client.4371.objecter ms_handle_connect 0x7f868006daa0 2024-03-29T11:40:51.570 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.568+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f869c0b9e40 con 0x7f86940c92b0 2024-03-29T11:40:51.570 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.568+0000 7f86997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:51.570 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.568+0000 7f86997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:51.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.954+0000 7f8686ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:51.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.954+0000 7f8686ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:51.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.954+0000 7f8686ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:40:51.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.954+0000 7f8686ffd640 1 -- 172.21.15.100:0/2749673749 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f867c005330 con 0x7f86940c92b0 2024-03-29T11:40:51.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.955+0000 7f86997fa640 1 -- 172.21.15.100:0/2749673749 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f869c0bbd70 con 0x7f86940c92b0 2024-03-29T11:40:51.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.955+0000 7f86997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:51.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.955+0000 7f86997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:51.958 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:40:51.958 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:51.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.958+0000 7f8686ffd640 10 librados: watch_flush enter 2024-03-29T11:40:51.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.958+0000 7f8686ffd640 10 librados: watch_flush exit 2024-03-29T11:40:51.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.958+0000 7f8686ffd640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-29T11:40:51.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.958+0000 7f8686ffd640 10 client.4371.objecter successfully canceled tick 2024-03-29T11:40:51.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.959+0000 7f8686ffd640 1 -- 172.21.15.100:0/2749673749 >> v1:172.21.15.100:6816/3220289657 conn(0x7f868006daa0 legacy=0x7f868006ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:51.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.959+0000 7f8686ffd640 10 monclient: shutdown 2024-03-29T11:40:51.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.959+0000 7f8686ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:51.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.959+0000 7f8686ffd640 1 -- 172.21.15.100:0/2749673749 >> v1:172.21.15.100:6789/0 conn(0x7f86940c92b0 legacy=0x7f8694031e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:51.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.959+0000 7f8686ffd640 1 -- 172.21.15.100:0/2749673749 shutdown_connections 2024-03-29T11:40:51.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.959+0000 7f8686ffd640 1 -- 172.21.15.100:0/2749673749 >> 172.21.15.100:0/2749673749 conn(0x7f86940058f0 msgr2=0x7f8694034490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:51.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.960+0000 7f8686ffd640 1 -- 172.21.15.100:0/2749673749 shutdown_connections 2024-03-29T11:40:51.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.962+0000 7f8686ffd640 1 -- 172.21.15.100:0/2749673749 wait complete. 2024-03-29T11:40:51.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:51.962+0000 7f8686ffd640 1 librados: shutdown 2024-03-29T11:40:51.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:58.788 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:58.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.927+0000 7f6051855640 10 monclient: get_monmap_and_config 2024-03-29T11:40:58.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.927+0000 7f6051855640 10 monclient: build_initial_monmap 2024-03-29T11:40:58.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.927+0000 7f6051855640 10 monclient: monmap: 2024-03-29T11:40:58.929 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:58.930 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:58.930 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:40:58.928157+0000 2024-03-29T11:40:58.930 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:40:58.928157+0000 2024-03-29T11:40:58.930 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:58.930 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:58.930 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:58.930 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:40:58.930 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:40:58.930 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:58.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.928+0000 7f6051855640 1 Processor -- start 2024-03-29T11:40:58.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.928+0000 7f6051855640 1 -- start start 2024-03-29T11:40:58.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.928+0000 7f6051855640 10 monclient: init 2024-03-29T11:40:58.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f6051855640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:58.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f6051855640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:58.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f6051855640 10 monclient(hunting): picked mon.noname-b con 0x7f604c1422b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:58.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f6051855640 10 monclient(hunting): picked mon.noname-a con 0x7f604c142690 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:58.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f6051855640 10 monclient(hunting): picked mon.noname-c con 0x7f604c0b17d0 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:58.932 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f6051855640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c142c80 con 0x7f604c142690 2024-03-29T11:40:58.932 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f6051855640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c0aae40 con 0x7f604c0b17d0 2024-03-29T11:40:58.932 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f6051855640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c13fbd0 con 0x7f604c1422b0 2024-03-29T11:40:58.932 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f6051855640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:58.932 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f6051855640 10 monclient(hunting): authenticate will time out at 1179.124639s 2024-03-29T11:40:58.932 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f604a7fc640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f604c142690 0x7f604c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:56548/0 (socket says 172.21.15.100:56548) 2024-03-29T11:40:58.932 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.929+0000 7f604a7fc640 1 -- 172.21.15.100:0/4038748939 learned_addr learned my addr 172.21.15.100:0/4038748939 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:58.943 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061707169 0 0) 0x7f604c142c80 con 0x7f604c142690 2024-03-29T11:40:58.943 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:58.943 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:58.943 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:58.943 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6038003520 con 0x7f604c142690 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850783638 0 0) 0x7f604c0aae40 con 0x7f604c0b17d0 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604c142c80 con 0x7f604c0b17d0 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450984221 0 0) 0x7f604c13fbd0 con 0x7f604c1422b0 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604c0aae40 con 0x7f604c1422b0 2024-03-29T11:40:58.944 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1968178064 0 0) 0x7f6038003520 con 0x7f604c142690 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 >> v1:172.21.15.100:6790/0 conn(0x7f604c1422b0 legacy=0x7f604c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 >> v1:172.21.15.191:6789/0 conn(0x7f604c0b17d0 legacy=0x7f604c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 monclient: found mon.noname-a 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c151940 con 0x7f604c142690 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.930+0000 7f6049ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6051855640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6051855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f603c001540 con 0x7f604c142690 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6049ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:58.945 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6049ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6049ffb640 10 monclient: dump: 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6051855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f603c001760 con 0x7f604c142690 2024-03-29T11:40:58.946 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6049ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6051855640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6051855640 4 set_mon_vals no callback set 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6049ffb640 1 -- 172.21.15.100:0/4038748939 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f603c001af0 con 0x7f604c142690 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6049ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6049ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.931+0000 7f6049ffb640 10 monclient: dump: 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:58.947 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.932+0000 7f6051855640 10 monclient: shutdown 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.932+0000 7f6051855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.932+0000 7f6051855640 1 -- 172.21.15.100:0/4038748939 >> v1:172.21.15.100:6789/0 conn(0x7f604c142690 legacy=0x7f604c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.932+0000 7f6051855640 1 -- 172.21.15.100:0/4038748939 shutdown_connections 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.932+0000 7f6051855640 1 -- 172.21.15.100:0/4038748939 >> 172.21.15.100:0/4038748939 conn(0x7f604c066a00 msgr2=0x7f604c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.932+0000 7f6051855640 1 -- 172.21.15.100:0/4038748939 shutdown_connections 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.932+0000 7f6051855640 1 -- 172.21.15.100:0/4038748939 wait complete. 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.933+0000 7f6051855640 10 monclient: build_initial_monmap 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.933+0000 7f6051855640 10 monclient: monmap: 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:58.948 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.933+0000 7f6051855640 1 librados: starting msgr at 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.933+0000 7f6051855640 1 librados: starting objecter 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.934+0000 7f6051855640 1 Processor -- start 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.934+0000 7f6051855640 1 -- start start 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.934+0000 7f6051855640 1 librados: setting wanted keys 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.934+0000 7f6051855640 1 librados: calling monclient init 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.934+0000 7f6051855640 10 monclient: init 2024-03-29T11:40:58.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.935+0000 7f6051855640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.935+0000 7f6051855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.935+0000 7f6051855640 10 monclient(hunting): picked mon.noname-b con 0x7f604c0b17d0 addr v1:172.21.15.191:6789/0 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.935+0000 7f6051855640 10 monclient(hunting): picked mon.noname-a con 0x7f604c1422b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.935+0000 7f6051855640 10 monclient(hunting): picked mon.noname-c con 0x7f604c142690 addr v1:172.21.15.100:6790/0 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.935+0000 7f6051855640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c0acb60 con 0x7f604c1422b0 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.935+0000 7f6051855640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c0ace40 con 0x7f604c0b17d0 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.935+0000 7f6051855640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c0ad120 con 0x7f604c142690 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.935+0000 7f6051855640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f604a7fc640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f604c1422b0 0x7f604c12b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:56874/0 (socket says 172.21.15.100:56874) 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f604a7fc640 1 -- 172.21.15.100:0/502635371 learned_addr learned my addr 172.21.15.100:0/502635371 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6051855640 10 monclient(hunting): authenticate will time out at 1179.131308s 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 client.?.objecter ms_handle_connect 0x7f604c1422b0 2024-03-29T11:40:58.950 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 client.?.objecter ms_handle_connect 0x7f604c142690 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633696442 0 0) 0x7f604c0acb60 con 0x7f604c1422b0 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f602c004eb0 con 0x7f604c1422b0 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129841244 0 0) 0x7f604c0ad120 con 0x7f604c142690 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.936+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604c0acb60 con 0x7f604c142690 2024-03-29T11:40:58.951 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1108868214 0 0) 0x7f604c0acb60 con 0x7f604c142690 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f604c0ad120 con 0x7f604c142690 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6034001930 con 0x7f604c142690 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 10 monclient(hunting): dump: 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:58.952 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 575268145 0 0) 0x7f604c0ad120 con 0x7f604c142690 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 >> v1:172.21.15.191:6789/0 conn(0x7f604c0b17d0 legacy=0x7f604c071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 >> v1:172.21.15.100:6789/0 conn(0x7f604c1422b0 legacy=0x7f604c12b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 1 monclient: found mon.c 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c0ab6e0 con 0x7f604c142690 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 10 monclient: _finish_auth 0 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6043fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6051855640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:40:58.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6051855640 10 monclient: _renew_subs 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6051855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6051855640 1 -- 172.21.15.100:0/502635371 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c0b0670 con 0x7f604c142690 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6051855640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6051855640 10 monclient: _renew_subs 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6051855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6051855640 1 -- 172.21.15.100:0/502635371 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c0b0b00 con 0x7f604c142690 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.937+0000 7f6051855640 1 librados: init done 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.939+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6034003610 con 0x7f604c142690 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.939+0000 7f6043fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.939+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6034001d80 con 0x7f604c142690 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.939+0000 7f6043fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.939+0000 7f6043fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:58.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.939+0000 7f6043fff640 10 monclient: dump: 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.939+0000 7f60497fa640 4 set_mon_vals no callback set 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.940+0000 7f6051855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.940+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f6034003aa0 con 0x7f604c142690 2024-03-29T11:40:58.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.941+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f60340837d0 con 0x7f604c142690 2024-03-29T11:40:58.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.941+0000 7f6043fff640 10 client.4409.objecter ms_dispatch 0x7f604c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:58.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.941+0000 7f6043fff640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:58.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.941+0000 7f6043fff640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:58.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.941+0000 7f6043fff640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:40:58.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.942+0000 7f6051855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:58.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.942+0000 7f6051855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:58.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.942+0000 7f6051855640 1 -- 172.21.15.100:0/502635371 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c0b4ae0 con 0x7f604c142690 2024-03-29T11:40:58.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.942+0000 7f604affd640 10 monclient: get_auth_request con 0x7f602c06dd30 auth_method 0 2024-03-29T11:40:58.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.942+0000 7f6043fff640 10 client.4409.objecter ms_handle_connect 0x7f602c06dd30 2024-03-29T11:40:58.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.956+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f603404f0b0 con 0x7f604c142690 2024-03-29T11:40:58.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.956+0000 7f6043fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:58.958 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:58.956+0000 7f6043fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:59.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.308+0000 7f6051855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:59.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.308+0000 7f6051855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:59.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.308+0000 7f6051855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:40:59.310 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.308+0000 7f6051855640 1 -- 172.21.15.100:0/502635371 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f604c0b4520 con 0x7f604c142690 2024-03-29T11:40:59.311 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.309+0000 7f6043fff640 1 -- 172.21.15.100:0/502635371 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f6034050fe0 con 0x7f604c142690 2024-03-29T11:40:59.311 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.309+0000 7f6043fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:59.311 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.309+0000 7f6043fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:59.311 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:40:59.312 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:59.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.312+0000 7f6051855640 10 librados: watch_flush enter 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 10 librados: watch_flush exit 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 1 -- 172.21.15.100:0/502635371 >> v1:172.21.15.100:6816/3220289657 conn(0x7f602c06dd30 legacy=0x7f602c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 10 monclient: shutdown 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 1 -- 172.21.15.100:0/502635371 >> v1:172.21.15.100:6790/0 conn(0x7f604c142690 legacy=0x7f604c12b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 1 -- 172.21.15.100:0/502635371 shutdown_connections 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 1 -- 172.21.15.100:0/502635371 >> 172.21.15.100:0/502635371 conn(0x7f604c066a00 msgr2=0x7f604c13db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 1 -- 172.21.15.100:0/502635371 shutdown_connections 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 1 -- 172.21.15.100:0/502635371 wait complete. 2024-03-29T11:40:59.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:40:59.313+0000 7f6051855640 1 librados: shutdown 2024-03-29T11:40:59.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:06.136 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:06.266 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.264+0000 7f309ccb3640 10 monclient: get_monmap_and_config 2024-03-29T11:41:06.266 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.264+0000 7f309ccb3640 10 monclient: build_initial_monmap 2024-03-29T11:41:06.266 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.264+0000 7f309ccb3640 10 monclient: monmap: 2024-03-29T11:41:06.267 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:06.267 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:06.267 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:41:06.265354+0000 2024-03-29T11:41:06.267 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:41:06.265354+0000 2024-03-29T11:41:06.267 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:06.267 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:06.267 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:06.267 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:41:06.267 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:41:06.267 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:06.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.265+0000 7f309ccb3640 1 Processor -- start 2024-03-29T11:41:06.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.265+0000 7f309ccb3640 1 -- start start 2024-03-29T11:41:06.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.265+0000 7f309ccb3640 10 monclient: init 2024-03-29T11:41:06.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f309ccb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:06.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f309ccb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f309ccb3640 10 monclient(hunting): picked mon.noname-b con 0x7f30980b4920 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f309ccb3640 10 monclient(hunting): picked mon.noname-c con 0x7f30980b4200 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f309ccb3640 10 monclient(hunting): picked mon.noname-a con 0x7f309813f9a0 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f309ccb3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30980b0930 con 0x7f309813f9a0 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f309ccb3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30980af460 con 0x7f30980b4200 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f309ccb3640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30980aeda0 con 0x7f30980b4920 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f309ccb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f309ccb3640 10 monclient(hunting): authenticate will time out at 1186.461475s 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f30977fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f30980b4920 0x7f309813f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:35040/0 (socket says 172.21.15.100:35040) 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f30977fe640 1 -- 172.21.15.100:0/954037341 learned_addr learned my addr 172.21.15.100:0/954037341 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f30967fc640 1 -- 172.21.15.100:0/954037341 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396221664 0 0) 0x7f30980aeda0 con 0x7f30980b4920 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f30967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f30967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:06.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f30967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:06.271 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f30967fc640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:41:06.271 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.266+0000 7f30967fc640 1 -- 172.21.15.100:0/954037341 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3080003520 con 0x7f30980b4920 2024-03-29T11:41:06.271 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.267+0000 7f30967fc640 1 -- 172.21.15.100:0/954037341 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 69759368 0 0) 0x7f3080003520 con 0x7f30980b4920 2024-03-29T11:41:06.271 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.267+0000 7f30967fc640 1 -- 172.21.15.100:0/954037341 >> v1:172.21.15.191:6789/0 conn(0x7f30980b4200 legacy=0x7f30980b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:06.271 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.267+0000 7f30967fc640 1 -- 172.21.15.100:0/954037341 >> v1:172.21.15.100:6789/0 conn(0x7f309813f9a0 legacy=0x7f309814d8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:06.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:06.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 1 monclient: found mon.noname-b 2024-03-29T11:41:06.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:06.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:41:06.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 1 -- 172.21.15.100:0/954037341 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30980af980 con 0x7f30980b4920 2024-03-29T11:41:06.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 1 -- 172.21.15.100:0/954037341 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3090001690 con 0x7f30980b4920 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f30967fc640 10 monclient: dump: 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:06.273 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f309ccb3640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.269+0000 7f309ccb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.270+0000 7f30967fc640 1 -- 172.21.15.100:0/954037341 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30900018f0 con 0x7f30980b4920 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.270+0000 7f30967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.270+0000 7f30967fc640 1 -- 172.21.15.100:0/954037341 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3090001d40 con 0x7f30980b4920 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.270+0000 7f30967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.270+0000 7f30967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.270+0000 7f30967fc640 10 monclient: dump: 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:06.274 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:06.275 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:06.275 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:06.275 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:06.275 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:06.275 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:06.275 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:06.275 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:06.275 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:06.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.270+0000 7f309ccb3640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:06.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.270+0000 7f309ccb3640 4 set_mon_vals no callback set 2024-03-29T11:41:06.282 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.271+0000 7f309ccb3640 10 monclient: shutdown 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.271+0000 7f309ccb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.271+0000 7f309ccb3640 1 -- 172.21.15.100:0/954037341 >> v1:172.21.15.100:6790/0 conn(0x7f30980b4920 legacy=0x7f309813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.271+0000 7f309ccb3640 1 -- 172.21.15.100:0/954037341 shutdown_connections 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.271+0000 7f309ccb3640 1 -- 172.21.15.100:0/954037341 >> 172.21.15.100:0/954037341 conn(0x7f309806d230 msgr2=0x7f3098066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.271+0000 7f309ccb3640 1 -- 172.21.15.100:0/954037341 shutdown_connections 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.271+0000 7f309ccb3640 1 -- 172.21.15.100:0/954037341 wait complete. 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.272+0000 7f309ccb3640 10 monclient: build_initial_monmap 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.272+0000 7f309ccb3640 10 monclient: monmap: 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:41:06.283 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.272+0000 7f309ccb3640 1 librados: starting msgr at 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.272+0000 7f309ccb3640 1 librados: starting objecter 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.272+0000 7f309ccb3640 1 Processor -- start 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.273+0000 7f309ccb3640 1 -- start start 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.273+0000 7f309ccb3640 1 librados: setting wanted keys 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.273+0000 7f309ccb3640 1 librados: calling monclient init 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.273+0000 7f309ccb3640 10 monclient: init 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f309ccb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:06.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f309ccb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f309ccb3640 10 monclient(hunting): picked mon.noname-a con 0x7f30980b4200 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f309ccb3640 10 monclient(hunting): picked mon.noname-b con 0x7f30980b4920 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f309ccb3640 10 monclient(hunting): picked mon.noname-c con 0x7f309813f9a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f309ccb3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3098139670 con 0x7f30980b4200 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f309ccb3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3098146160 con 0x7f30980b4920 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f309ccb3640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30981462f0 con 0x7f309813f9a0 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f309ccb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f309ccb3640 10 monclient(hunting): authenticate will time out at 1186.469217s 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f3097fff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f309813f9a0 0x7f309813a2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:35748/0 (socket says 172.21.15.100:35748) 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f3097fff640 1 -- 172.21.15.100:0/1405251406 learned_addr learned my addr 172.21.15.100:0/1405251406 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f307ffff640 10 client.?.objecter ms_handle_connect 0x7f309813f9a0 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f307ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:06.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646684850 0 0) 0x7f30981462f0 con 0x7f309813f9a0 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f307ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f307ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f307ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f307ffff640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3084004eb0 con 0x7f309813f9a0 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 350169925 0 0) 0x7f3084004eb0 con 0x7f309813f9a0 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.274+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3084005300 con 0x7f309813f9a0 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.275+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3088001a90 con 0x7f309813f9a0 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.275+0000 7f307ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.275+0000 7f307ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.275+0000 7f307ffff640 10 monclient(hunting): dump: 2024-03-29T11:41:06.286 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.275+0000 7f307ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.275+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3101878373 0 0) 0x7f3084005300 con 0x7f309813f9a0 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.275+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 >> v1:172.21.15.191:6789/0 conn(0x7f30980b4920 legacy=0x7f3098139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.275+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 >> v1:172.21.15.100:6789/0 conn(0x7f30980b4200 legacy=0x7f3098145a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:06.287 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 1 monclient: found mon.c 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3098144d40 con 0x7f309813f9a0 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 10 monclient: _finish_auth 0 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f309ccb3640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f309ccb3640 10 monclient: _renew_subs 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f309ccb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f309ccb3640 1 -- 172.21.15.100:0/1405251406 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3098144fb0 con 0x7f309813f9a0 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f309ccb3640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:06.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f309ccb3640 10 monclient: _renew_subs 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f309ccb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f309ccb3640 1 -- 172.21.15.100:0/1405251406 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30981ef500 con 0x7f309813f9a0 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f309ccb3640 1 librados: init done 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3088002060 con 0x7f309813f9a0 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3088001a90 con 0x7f309813f9a0 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f307ffff640 10 monclient: dump: 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:06.289 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.277+0000 7f3095ffb640 4 set_mon_vals no callback set 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.279+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f308801aae0 con 0x7f309813f9a0 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.280+0000 7f30977fe640 10 monclient: get_auth_request con 0x7f308406dee0 auth_method 0 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.280+0000 7f307ffff640 10 client.4415.objecter ms_handle_connect 0x7f308406dee0 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.280+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f30880548b0 con 0x7f309813f9a0 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.280+0000 7f307ffff640 10 client.4415.objecter ms_dispatch 0x7f3098000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:06.290 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.280+0000 7f307ffff640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:06.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.280+0000 7f307ffff640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:06.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.280+0000 7f307ffff640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T11:41:06.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.281+0000 7f309ccb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:06.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.281+0000 7f309ccb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:06.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.281+0000 7f309ccb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:06.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.281+0000 7f309ccb3640 1 -- 172.21.15.100:0/1405251406 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3078005080 con 0x7f309813f9a0 2024-03-29T11:41:06.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.289+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3088050b50 con 0x7f309813f9a0 2024-03-29T11:41:06.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.289+0000 7f307ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:06.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.289+0000 7f307ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:06.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.599+0000 7f309ccb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:06.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.599+0000 7f309ccb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:06.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.599+0000 7f309ccb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:06.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.599+0000 7f309ccb3640 1 -- 172.21.15.100:0/1405251406 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3078005330 con 0x7f309813f9a0 2024-03-29T11:41:06.602 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.600+0000 7f307ffff640 1 -- 172.21.15.100:0/1405251406 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f3088050cf0 con 0x7f309813f9a0 2024-03-29T11:41:06.602 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.600+0000 7f307ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:06.602 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.600+0000 7f307ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:06.603 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:41:06.603 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:06.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.604+0000 7f309ccb3640 10 librados: watch_flush enter 2024-03-29T11:41:06.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.604+0000 7f309ccb3640 10 librados: watch_flush exit 2024-03-29T11:41:06.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.604+0000 7f309ccb3640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T11:41:06.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.604+0000 7f309ccb3640 10 client.4415.objecter successfully canceled tick 2024-03-29T11:41:06.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.604+0000 7f309ccb3640 1 -- 172.21.15.100:0/1405251406 >> v1:172.21.15.100:6816/3220289657 conn(0x7f308406dee0 legacy=0x7f30840703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:06.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.604+0000 7f309ccb3640 10 monclient: shutdown 2024-03-29T11:41:06.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.604+0000 7f309ccb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:06.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.604+0000 7f309ccb3640 1 -- 172.21.15.100:0/1405251406 >> v1:172.21.15.100:6790/0 conn(0x7f309813f9a0 legacy=0x7f309813a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:06.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.604+0000 7f309ccb3640 1 -- 172.21.15.100:0/1405251406 shutdown_connections 2024-03-29T11:41:06.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.604+0000 7f309ccb3640 1 -- 172.21.15.100:0/1405251406 >> 172.21.15.100:0/1405251406 conn(0x7f309806d230 msgr2=0x7f30980b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:06.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.605+0000 7f309ccb3640 1 -- 172.21.15.100:0/1405251406 shutdown_connections 2024-03-29T11:41:06.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.605+0000 7f309ccb3640 1 -- 172.21.15.100:0/1405251406 wait complete. 2024-03-29T11:41:06.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:06.605+0000 7f309ccb3640 1 librados: shutdown 2024-03-29T11:41:06.622 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:13.428 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:13.577 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.575+0000 7f89dffff640 10 monclient: get_monmap_and_config 2024-03-29T11:41:13.577 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.575+0000 7f89dffff640 10 monclient: build_initial_monmap 2024-03-29T11:41:13.577 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.575+0000 7f89dffff640 10 monclient: monmap: 2024-03-29T11:41:13.577 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:13.577 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:13.577 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:41:13.576215+0000 2024-03-29T11:41:13.578 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:41:13.576215+0000 2024-03-29T11:41:13.578 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:13.578 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:13.578 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:13.578 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:41:13.578 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:41:13.578 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:13.578 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.576+0000 7f89dffff640 1 Processor -- start 2024-03-29T11:41:13.579 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.576+0000 7f89dffff640 1 -- start start 2024-03-29T11:41:13.579 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.576+0000 7f89dffff640 10 monclient: init 2024-03-29T11:41:13.579 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.577+0000 7f89dffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:13.579 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.577+0000 7f89dffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:13.579 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.577+0000 7f89dffff640 10 monclient(hunting): picked mon.noname-c con 0x7f89e00b2800 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:13.584 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.580+0000 7f89deffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f89e00b2800 0x7f89e0142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:53117/0 (socket says 172.21.15.100:53117) 2024-03-29T11:41:13.584 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.581+0000 7f89dffff640 10 monclient(hunting): picked mon.noname-a con 0x7f89e00a9c00 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:13.584 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.581+0000 7f89deffd640 1 -- 172.21.15.100:0/1160816707 learned_addr learned my addr 172.21.15.100:0/1160816707 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:13.584 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.581+0000 7f89dffff640 10 monclient(hunting): picked mon.noname-b con 0x7f89e00b38c0 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:13.584 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.581+0000 7f89dffff640 1 -- 172.21.15.100:0/1160816707 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e0141eb0 con 0x7f89e00a9c00 2024-03-29T11:41:13.584 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.581+0000 7f89dffff640 1 -- 172.21.15.100:0/1160816707 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e00b22d0 con 0x7f89e00b2800 2024-03-29T11:41:13.584 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.581+0000 7f89dffff640 1 -- 172.21.15.100:0/1160816707 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e0142d30 con 0x7f89e00b38c0 2024-03-29T11:41:13.584 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.581+0000 7f89dffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:13.584 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.581+0000 7f89dffff640 10 monclient(hunting): authenticate will time out at 1193.776669s 2024-03-29T11:41:13.585 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 1 -- 172.21.15.100:0/1160816707 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709018584 0 0) 0x7f89e0142d30 con 0x7f89e00b38c0 2024-03-29T11:41:13.585 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:13.585 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:13.585 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:13.585 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:41:13.585 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 1 -- 172.21.15.100:0/1160816707 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89c8003520 con 0x7f89e00b38c0 2024-03-29T11:41:13.585 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 1 -- 172.21.15.100:0/1160816707 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1871707859 0 0) 0x7f89c8003520 con 0x7f89e00b38c0 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 1 -- 172.21.15.100:0/1160816707 >> v1:172.21.15.191:6789/0 conn(0x7f89e00b2800 legacy=0x7f89e0142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 1 -- 172.21.15.100:0/1160816707 >> v1:172.21.15.100:6789/0 conn(0x7f89e00a9c00 legacy=0x7f89e013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 1 monclient: found mon.noname-b 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 1 -- 172.21.15.100:0/1160816707 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e00aaf90 con 0x7f89e00b38c0 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89ddffb640 1 -- 172.21.15.100:0/1160816707 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f89d8001940 con 0x7f89e00b38c0 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89dffff640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.582+0000 7f89dffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:13.586 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89ddffb640 10 monclient: dump: 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:13.587 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89dffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89ddffb640 1 -- 172.21.15.100:0/1160816707 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89d8001aa0 con 0x7f89e00b38c0 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89ddffb640 1 -- 172.21.15.100:0/1160816707 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f89d8001e70 con 0x7f89e00b38c0 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89ddffb640 10 monclient: dump: 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:13.589 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89dffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.583+0000 7f89dffff640 4 set_mon_vals no callback set 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.584+0000 7f89dffff640 10 monclient: shutdown 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.584+0000 7f89dffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.584+0000 7f89dffff640 1 -- 172.21.15.100:0/1160816707 >> v1:172.21.15.100:6790/0 conn(0x7f89e00b38c0 legacy=0x7f89e014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.586+0000 7f89dffff640 1 -- 172.21.15.100:0/1160816707 shutdown_connections 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.586+0000 7f89dffff640 1 -- 172.21.15.100:0/1160816707 >> 172.21.15.100:0/1160816707 conn(0x7f89e0066a00 msgr2=0x7f89e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:13.590 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.587+0000 7f89dffff640 1 -- 172.21.15.100:0/1160816707 shutdown_connections 2024-03-29T11:41:13.591 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.587+0000 7f89dffff640 1 -- 172.21.15.100:0/1160816707 wait complete. 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.588+0000 7f89dffff640 10 monclient: build_initial_monmap 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.588+0000 7f89dffff640 10 monclient: monmap: 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:13.592 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.588+0000 7f89dffff640 1 librados: starting msgr at 2024-03-29T11:41:13.593 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.588+0000 7f89dffff640 1 librados: starting objecter 2024-03-29T11:41:13.593 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.589+0000 7f89dffff640 1 Processor -- start 2024-03-29T11:41:13.593 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.589+0000 7f89dffff640 1 -- start start 2024-03-29T11:41:13.593 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.589+0000 7f89dffff640 1 librados: setting wanted keys 2024-03-29T11:41:13.593 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.589+0000 7f89dffff640 1 librados: calling monclient init 2024-03-29T11:41:13.593 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.589+0000 7f89dffff640 10 monclient: init 2024-03-29T11:41:13.594 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.590+0000 7f89dffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:13.594 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.590+0000 7f89dffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:13.594 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.591+0000 7f89dffff640 10 monclient(hunting): picked mon.noname-b con 0x7f89e00a9c00 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:13.594 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.591+0000 7f89dffff640 10 monclient(hunting): picked mon.noname-a con 0x7f89e00b2800 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:13.594 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.591+0000 7f89dffff640 10 monclient(hunting): picked mon.noname-c con 0x7f89e00b38c0 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.591+0000 7f89dffff640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e00abb60 con 0x7f89e00b2800 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.591+0000 7f89dffff640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e01d2ad0 con 0x7f89e00a9c00 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.591+0000 7f89dffff640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e006de90 con 0x7f89e00b38c0 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.591+0000 7f89dffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.591+0000 7f89dffff640 10 monclient(hunting): authenticate will time out at 1193.786450s 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.591+0000 7f89df7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f89e00b38c0 0x7f89e013b030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:58436/0 (socket says 172.21.15.100:58436) 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.591+0000 7f89df7fe640 1 -- 172.21.15.100:0/2968014298 learned_addr learned my addr 172.21.15.100:0/2968014298 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 10 client.?.objecter ms_handle_connect 0x7f89e00b38c0 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115667879 0 0) 0x7f89e006de90 con 0x7f89e00b38c0 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:13.595 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:41:13.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89cc004eb0 con 0x7f89e00b38c0 2024-03-29T11:41:13.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3335951479 0 0) 0x7f89cc004eb0 con 0x7f89e00b38c0 2024-03-29T11:41:13.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89e006de90 con 0x7f89e00b38c0 2024-03-29T11:41:13.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f89d8001e70 con 0x7f89e00b38c0 2024-03-29T11:41:13.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:13.596 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 10 monclient(hunting): dump: 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.592+0000 7f89c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.593+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1065419526 0 0) 0x7f89e006de90 con 0x7f89e00b38c0 2024-03-29T11:41:13.597 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.593+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 >> v1:172.21.15.191:6789/0 conn(0x7f89e00a9c00 legacy=0x7f89e01d2ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:13.598 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.594+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 >> v1:172.21.15.100:6789/0 conn(0x7f89e00b2800 legacy=0x7f89e013a920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:13.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.594+0000 7f89c77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 1 monclient: found mon.c 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e006e020 con 0x7f89e00b38c0 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89dffff640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89dffff640 10 monclient: _renew_subs 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89dffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89dffff640 1 -- 172.21.15.100:0/2968014298 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e01d3700 con 0x7f89e00b38c0 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89dffff640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:13.600 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89dffff640 10 monclient: _renew_subs 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89dffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89d8002040 con 0x7f89e00b38c0 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89dffff640 1 -- 172.21.15.100:0/2968014298 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e01d39b0 con 0x7f89e00b38c0 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89dffff640 1 librados: init done 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f89d8003e80 con 0x7f89e00b38c0 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89dd7fa640 4 set_mon_vals no callback set 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.595+0000 7f89c77fe640 10 monclient: dump: 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:13.601 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:13.602 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:13.602 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:13.602 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:13.602 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:13.602 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:13.602 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.599+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f89d8003fe0 con 0x7f89e00b38c0 2024-03-29T11:41:13.603 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.599+0000 7f89dffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:13.603 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.599+0000 7f89dffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:13.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.599+0000 7f89dffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:13.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.599+0000 7f89dffff640 1 -- 172.21.15.100:0/2968014298 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c0005080 con 0x7f89e00b38c0 2024-03-29T11:41:13.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.601+0000 7f89deffd640 10 monclient: get_auth_request con 0x7f89cc06dd00 auth_method 0 2024-03-29T11:41:13.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.601+0000 7f89c77fe640 10 client.4421.objecter ms_handle_connect 0x7f89cc06dd00 2024-03-29T11:41:13.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.601+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f89d8084ea0 con 0x7f89e00b38c0 2024-03-29T11:41:13.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.602+0000 7f89c77fe640 10 client.4421.objecter ms_dispatch 0x7f89e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:13.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.602+0000 7f89c77fe640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:13.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.602+0000 7f89c77fe640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:13.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.602+0000 7f89c77fe640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T11:41:13.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.607+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89d8050780 con 0x7f89e00b38c0 2024-03-29T11:41:13.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.607+0000 7f89c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:13.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.607+0000 7f89c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.919+0000 7f89dffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.919+0000 7f89dffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.919+0000 7f89dffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.919+0000 7f89dffff640 1 -- 172.21.15.100:0/2968014298 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89c0005330 con 0x7f89e00b38c0 2024-03-29T11:41:13.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.920+0000 7f89c77fe640 1 -- 172.21.15.100:0/2968014298 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f89d80526b0 con 0x7f89e00b38c0 2024-03-29T11:41:13.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.920+0000 7f89c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:13.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.920+0000 7f89c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:13.923 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:41:13.923 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 10 librados: watch_flush enter 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 10 librados: watch_flush exit 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 10 client.4421.objecter successfully canceled tick 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 1 -- 172.21.15.100:0/2968014298 >> v1:172.21.15.100:6816/3220289657 conn(0x7f89cc06dd00 legacy=0x7f89cc0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 10 monclient: shutdown 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 1 -- 172.21.15.100:0/2968014298 >> v1:172.21.15.100:6790/0 conn(0x7f89e00b38c0 legacy=0x7f89e013b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 1 -- 172.21.15.100:0/2968014298 shutdown_connections 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 1 -- 172.21.15.100:0/2968014298 >> 172.21.15.100:0/2968014298 conn(0x7f89e0066a00 msgr2=0x7f89e01404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 1 -- 172.21.15.100:0/2968014298 shutdown_connections 2024-03-29T11:41:13.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 1 -- 172.21.15.100:0/2968014298 wait complete. 2024-03-29T11:41:13.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:13.923+0000 7f89dffff640 1 librados: shutdown 2024-03-29T11:41:13.939 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:20.746 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:20.896 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.894+0000 7fa43a20e640 10 monclient: get_monmap_and_config 2024-03-29T11:41:20.896 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.894+0000 7fa43a20e640 10 monclient: build_initial_monmap 2024-03-29T11:41:20.896 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.894+0000 7fa43a20e640 10 monclient: monmap: 2024-03-29T11:41:20.896 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:20.896 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:20.896 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:41:20.895169+0000 2024-03-29T11:41:20.896 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:41:20.895169+0000 2024-03-29T11:41:20.897 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:20.897 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:20.897 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:20.897 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:41:20.897 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:41:20.897 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:20.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.897+0000 7fa43a20e640 1 Processor -- start 2024-03-29T11:41:20.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.897+0000 7fa43a20e640 1 -- start start 2024-03-29T11:41:20.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.897+0000 7fa43a20e640 10 monclient: init 2024-03-29T11:41:20.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.898+0000 7fa43a20e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:20.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.898+0000 7fa43a20e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa43a20e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4341457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa43a20e640 10 monclient(hunting): picked mon.noname-b con 0x7fa43413f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa43a20e640 10 monclient(hunting): picked mon.noname-c con 0x7fa4340a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa43a20e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4340aaef0 con 0x7fa4341457f0 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa43a20e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa434145bd0 con 0x7fa4340a9d70 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa43a20e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa434145eb0 con 0x7fa43413f650 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa43a20e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa43a20e640 10 monclient(hunting): authenticate will time out at 1201.094347s 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa432ffd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fa43413f650 0x7fa4340b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:50992/0 (socket says 172.21.15.100:50992) 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa432ffd640 1 -- 172.21.15.100:0/100905061 learned_addr learned my addr 172.21.15.100:0/100905061 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:20.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa4327fc640 1 -- 172.21.15.100:0/100905061 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325314902 0 0) 0x7fa434145eb0 con 0x7fa43413f650 2024-03-29T11:41:20.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa4327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:20.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa4327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:20.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa4327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:20.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa4327fc640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:41:20.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa4327fc640 1 -- 172.21.15.100:0/100905061 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa41c003520 con 0x7fa43413f650 2024-03-29T11:41:20.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa4327fc640 1 -- 172.21.15.100:0/100905061 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 625167308 0 0) 0x7fa41c003520 con 0x7fa43413f650 2024-03-29T11:41:20.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.899+0000 7fa4327fc640 1 -- 172.21.15.100:0/100905061 >> v1:172.21.15.191:6789/0 conn(0x7fa4340a9d70 legacy=0x7fa43414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:20.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 1 -- 172.21.15.100:0/100905061 >> v1:172.21.15.100:6789/0 conn(0x7fa4341457f0 legacy=0x7fa43413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:20.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:20.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 1 monclient: found mon.noname-b 2024-03-29T11:41:20.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:20.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:41:20.906 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 1 -- 172.21.15.100:0/100905061 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa434146340 con 0x7fa43413f650 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 1 -- 172.21.15.100:0/100905061 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa424001690 con 0x7fa43413f650 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: dump: 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:20.907 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:20.908 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:20.908 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:20.908 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:20.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa43a20e640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:41:20.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa43a20e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 1 -- 172.21.15.100:0/100905061 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa424001860 con 0x7fa43413f650 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 1 -- 172.21.15.100:0/100905061 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa424001c70 con 0x7fa43413f650 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa4327fc640 10 monclient: dump: 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:20.909 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa43a20e640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.904+0000 7fa43a20e640 4 set_mon_vals no callback set 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.905+0000 7fa43a20e640 10 monclient: shutdown 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.905+0000 7fa43a20e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.905+0000 7fa43a20e640 1 -- 172.21.15.100:0/100905061 >> v1:172.21.15.100:6790/0 conn(0x7fa43413f650 legacy=0x7fa4340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.905+0000 7fa43a20e640 1 -- 172.21.15.100:0/100905061 shutdown_connections 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.905+0000 7fa43a20e640 1 -- 172.21.15.100:0/100905061 >> 172.21.15.100:0/100905061 conn(0x7fa434066a00 msgr2=0x7fa434066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.905+0000 7fa43a20e640 1 -- 172.21.15.100:0/100905061 shutdown_connections 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.906+0000 7fa43a20e640 1 -- 172.21.15.100:0/100905061 wait complete. 2024-03-29T11:41:20.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.907+0000 7fa43a20e640 10 monclient: build_initial_monmap 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.907+0000 7fa43a20e640 10 monclient: monmap: 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:41:20.911 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:20.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.907+0000 7fa43a20e640 1 librados: starting msgr at 2024-03-29T11:41:20.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.907+0000 7fa43a20e640 1 librados: starting objecter 2024-03-29T11:41:20.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.908+0000 7fa43a20e640 1 Processor -- start 2024-03-29T11:41:20.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.908+0000 7fa43a20e640 1 -- start start 2024-03-29T11:41:20.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.908+0000 7fa43a20e640 1 librados: setting wanted keys 2024-03-29T11:41:20.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.908+0000 7fa43a20e640 1 librados: calling monclient init 2024-03-29T11:41:20.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.908+0000 7fa43a20e640 10 monclient: init 2024-03-29T11:41:20.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.909+0000 7fa43a20e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:20.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.909+0000 7fa43a20e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:20.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.909+0000 7fa43a20e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4340a9d70 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:20.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.909+0000 7fa43a20e640 10 monclient(hunting): picked mon.noname-c con 0x7fa43413f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:20.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.912+0000 7fa43a20e640 10 monclient(hunting): picked mon.noname-b con 0x7fa4341457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:20.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.913+0000 7fa432ffd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fa43413f650 0x7fa4341d7b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:51820/0 (socket says 172.21.15.100:51820) 2024-03-29T11:41:20.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.913+0000 7fa432ffd640 1 -- 172.21.15.100:0/703443892 learned_addr learned my addr 172.21.15.100:0/703443892 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:20.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa41bfff640 10 client.?.objecter ms_handle_connect 0x7fa43413f650 2024-03-29T11:41:20.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa41bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:20.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa43413b970 con 0x7fa4340a9d70 2024-03-29T11:41:20.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4340acee0 con 0x7fa4341457f0 2024-03-29T11:41:20.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4340ad1c0 con 0x7fa43413f650 2024-03-29T11:41:20.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa43a20e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:20.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa43a20e640 10 monclient(hunting): authenticate will time out at 1201.110664s 2024-03-29T11:41:20.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611304242 0 0) 0x7fa4340ad1c0 con 0x7fa43413f650 2024-03-29T11:41:20.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa41bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:20.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa41bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:20.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa41bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa41bfff640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.915+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa420004eb0 con 0x7fa43413f650 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 client.?.objecter ms_handle_connect 0x7fa4340a9d70 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4128135049 0 0) 0x7fa420004eb0 con 0x7fa43413f650 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4340ad1c0 con 0x7fa43413f650 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa424001b70 con 0x7fa43413f650 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 monclient(hunting): dump: 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:20.920 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763898241 0 0) 0x7fa43413b970 con 0x7fa4340a9d70 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa420004eb0 con 0x7fa4340a9d70 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3771915280 0 0) 0x7fa4340ad1c0 con 0x7fa43413f650 2024-03-29T11:41:20.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 >> v1:172.21.15.191:6789/0 conn(0x7fa4341457f0 legacy=0x7fa4341e1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 >> v1:172.21.15.100:6789/0 conn(0x7fa4340a9d70 legacy=0x7fa4341d7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.916+0000 7fa41bfff640 1 monclient: found mon.c 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4340ad4a0 con 0x7fa43413f650 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 10 monclient: _finish_auth 0 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa424004bf0 con 0x7fa43413f650 2024-03-29T11:41:20.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa424004f00 con 0x7fa43413f650 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa41bfff640 10 monclient: dump: 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:20.923 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:20.924 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:20.924 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:20.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa43a20e640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:41:20.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa43a20e640 10 monclient: _renew_subs 2024-03-29T11:41:20.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa43a20e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:20.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.917+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43413b370 con 0x7fa43413f650 2024-03-29T11:41:20.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.918+0000 7fa43a20e640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:20.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.918+0000 7fa43a20e640 10 monclient: _renew_subs 2024-03-29T11:41:20.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.918+0000 7fa43a20e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:20.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.918+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4340ab870 con 0x7fa43413f650 2024-03-29T11:41:20.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.918+0000 7fa43a20e640 1 librados: init done 2024-03-29T11:41:20.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.919+0000 7fa431ffb640 4 set_mon_vals no callback set 2024-03-29T11:41:20.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.920+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fa42401aed0 con 0x7fa43413f650 2024-03-29T11:41:20.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.920+0000 7fa43a20e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:20.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.920+0000 7fa43a20e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:20.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.920+0000 7fa43a20e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:20.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.920+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa414005080 con 0x7fa43413f650 2024-03-29T11:41:20.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.920+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fa424084ce0 con 0x7fa43413f650 2024-03-29T11:41:20.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.920+0000 7fa41bfff640 10 client.4427.objecter ms_dispatch 0x7fa434000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:20.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.920+0000 7fa41bfff640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:20.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.920+0000 7fa41bfff640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:20.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.920+0000 7fa41bfff640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T11:41:20.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.921+0000 7fa4337fe640 10 monclient: get_auth_request con 0x7fa42006dd10 auth_method 0 2024-03-29T11:41:20.927 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.921+0000 7fa41bfff640 10 client.4427.objecter ms_handle_connect 0x7fa42006dd10 2024-03-29T11:41:20.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.929+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4240505c0 con 0x7fa43413f650 2024-03-29T11:41:20.931 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.929+0000 7fa41bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:20.932 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:20.929+0000 7fa41bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.256+0000 7fa43a20e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.257+0000 7fa43a20e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.257+0000 7fa43a20e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.257+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa414005330 con 0x7fa43413f650 2024-03-29T11:41:21.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.259+0000 7fa41bfff640 1 -- 172.21.15.100:0/703443892 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fa4240524f0 con 0x7fa43413f650 2024-03-29T11:41:21.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.259+0000 7fa41bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:21.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.259+0000 7fa41bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:21.262 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:41:21.262 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:21.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.262+0000 7fa43a20e640 10 librados: watch_flush enter 2024-03-29T11:41:21.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.262+0000 7fa43a20e640 10 librados: watch_flush exit 2024-03-29T11:41:21.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.262+0000 7fa43a20e640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T11:41:21.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.262+0000 7fa43a20e640 10 client.4427.objecter successfully canceled tick 2024-03-29T11:41:21.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.263+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 >> v1:172.21.15.100:6816/3220289657 conn(0x7fa42006dd10 legacy=0x7fa4200701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:21.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.263+0000 7fa43a20e640 10 monclient: shutdown 2024-03-29T11:41:21.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.263+0000 7fa43a20e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:21.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.263+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 >> v1:172.21.15.100:6790/0 conn(0x7fa43413f650 legacy=0x7fa4341d7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:21.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.265+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 shutdown_connections 2024-03-29T11:41:21.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.265+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 >> 172.21.15.100:0/703443892 conn(0x7fa434066a00 msgr2=0x7fa4340b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:21.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.266+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 shutdown_connections 2024-03-29T11:41:21.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.266+0000 7fa43a20e640 1 -- 172.21.15.100:0/703443892 wait complete. 2024-03-29T11:41:21.271 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:21.266+0000 7fa43a20e640 1 librados: shutdown 2024-03-29T11:41:21.289 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:28.095 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:28.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.243+0000 7f8b4b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:41:28.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.243+0000 7f8b4b59e640 10 monclient: build_initial_monmap 2024-03-29T11:41:28.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.243+0000 7f8b4b59e640 10 monclient: monmap: 2024-03-29T11:41:28.246 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:28.246 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:28.246 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:41:28.244574+0000 2024-03-29T11:41:28.246 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:41:28.244574+0000 2024-03-29T11:41:28.246 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:28.246 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:28.247 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:28.247 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:41:28.247 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:41:28.247 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:28.248 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.246+0000 7f8b4b59e640 1 Processor -- start 2024-03-29T11:41:28.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.247+0000 7f8b4b59e640 1 -- start start 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.247+0000 7f8b4b59e640 10 monclient: init 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.248+0000 7f8b4b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.248+0000 7f8b4b59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.248+0000 7f8b4b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f8b44038c60 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.248+0000 7f8b4b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8b440324b0 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.248+0000 7f8b4b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8b440d11e0 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.248+0000 7f8b4b59e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440caf80 con 0x7f8b440d11e0 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.248+0000 7f8b4b59e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b44039850 con 0x7f8b440324b0 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.248+0000 7f8b4b59e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440c78d0 con 0x7f8b44038c60 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.248+0000 7f8b4b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:28.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.248+0000 7f8b4b59e640 10 monclient(hunting): authenticate will time out at 1208.444001s 2024-03-29T11:41:28.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.249+0000 7f8b4a59c640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f8b44038c60 0x7f8b440c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:44732/0 (socket says 172.21.15.100:44732) 2024-03-29T11:41:28.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.249+0000 7f8b4a59c640 1 -- 172.21.15.100:0/1020172469 learned_addr learned my addr 172.21.15.100:0/1020172469 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:28.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.251+0000 7f8b4959a640 1 -- 172.21.15.100:0/1020172469 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748191302 0 0) 0x7f8b440c78d0 con 0x7f8b44038c60 2024-03-29T11:41:28.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.251+0000 7f8b4959a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:28.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.251+0000 7f8b4959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:28.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.251+0000 7f8b4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:28.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.251+0000 7f8b4959a640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:41:28.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.251+0000 7f8b4959a640 1 -- 172.21.15.100:0/1020172469 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b34003520 con 0x7f8b44038c60 2024-03-29T11:41:28.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.251+0000 7f8b4959a640 1 -- 172.21.15.100:0/1020172469 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4261241357 0 0) 0x7f8b34003520 con 0x7f8b44038c60 2024-03-29T11:41:28.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.251+0000 7f8b4959a640 1 -- 172.21.15.100:0/1020172469 >> v1:172.21.15.191:6789/0 conn(0x7f8b440324b0 legacy=0x7f8b440cb4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:28.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 1 -- 172.21.15.100:0/1020172469 >> v1:172.21.15.100:6789/0 conn(0x7f8b440d11e0 legacy=0x7f8b440d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:28.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 10 monclient: _finish_hunting 0 2024-03-29T11:41:28.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 1 monclient: found mon.noname-b 2024-03-29T11:41:28.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:28.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:41:28.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 1 -- 172.21.15.100:0/1020172469 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b440c7a60 con 0x7f8b44038c60 2024-03-29T11:41:28.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 10 monclient: _finish_auth 0 2024-03-29T11:41:28.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 10 monclient: _check_auth_tickets 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4b59e640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 1 -- 172.21.15.100:0/1020172469 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8b3c001690 con 0x7f8b44038c60 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4959a640 10 monclient: dump: 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:28.261 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.255+0000 7f8b4b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.256+0000 7f8b4959a640 1 -- 172.21.15.100:0/1020172469 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b3c001860 con 0x7f8b44038c60 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.256+0000 7f8b4959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.256+0000 7f8b4b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.256+0000 7f8b4b59e640 4 set_mon_vals no callback set 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.257+0000 7f8b4b59e640 10 monclient: shutdown 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.257+0000 7f8b4b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:28.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.257+0000 7f8b4b59e640 1 -- 172.21.15.100:0/1020172469 >> v1:172.21.15.100:6790/0 conn(0x7f8b44038c60 legacy=0x7f8b440c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:28.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.257+0000 7f8b4b59e640 1 -- 172.21.15.100:0/1020172469 shutdown_connections 2024-03-29T11:41:28.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.257+0000 7f8b4b59e640 1 -- 172.21.15.100:0/1020172469 >> 172.21.15.100:0/1020172469 conn(0x7f8b44005c20 msgr2=0x7f8b44006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.259+0000 7f8b4b59e640 1 -- 172.21.15.100:0/1020172469 shutdown_connections 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.259+0000 7f8b4b59e640 1 -- 172.21.15.100:0/1020172469 wait complete. 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.260+0000 7f8b4b59e640 10 monclient: build_initial_monmap 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.260+0000 7f8b4b59e640 10 monclient: monmap: 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:28.264 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:28.265 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:41:28.265 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:41:28.265 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:28.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.260+0000 7f8b4b59e640 1 librados: starting msgr at 2024-03-29T11:41:28.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.260+0000 7f8b4b59e640 1 librados: starting objecter 2024-03-29T11:41:28.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.260+0000 7f8b4b59e640 1 Processor -- start 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.261+0000 7f8b4b59e640 1 -- start start 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.261+0000 7f8b4b59e640 1 librados: setting wanted keys 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.261+0000 7f8b4b59e640 1 librados: calling monclient init 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.261+0000 7f8b4b59e640 10 monclient: init 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.262+0000 7f8b4b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.262+0000 7f8b4b59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.262+0000 7f8b4b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8b440324b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.262+0000 7f8b4b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8b44038c60 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.262+0000 7f8b4b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f8b440d11e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.262+0000 7f8b4b59e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440363b0 con 0x7f8b440324b0 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.262+0000 7f8b4b59e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b44036620 con 0x7f8b440d11e0 2024-03-29T11:41:28.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.262+0000 7f8b4b59e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b44036900 con 0x7f8b44038c60 2024-03-29T11:41:28.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.262+0000 7f8b4b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:28.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.262+0000 7f8b4b59e640 10 monclient(hunting): authenticate will time out at 1208.457184s 2024-03-29T11:41:28.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.264+0000 7f8b4ad9d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f8b440d11e0 0x7f8b4416b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:58247/0 (socket says 172.21.15.100:58247) 2024-03-29T11:41:28.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.264+0000 7f8b4ad9d640 1 -- 172.21.15.100:0/3435148383 learned_addr learned my addr 172.21.15.100:0/3435148383 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:28.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 client.?.objecter ms_handle_connect 0x7f8b440324b0 2024-03-29T11:41:28.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:28.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 client.?.objecter ms_handle_connect 0x7f8b44038c60 2024-03-29T11:41:28.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:28.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 client.?.objecter ms_handle_connect 0x7f8b440d11e0 2024-03-29T11:41:28.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:28.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220813163 0 0) 0x7f8b44036900 con 0x7f8b44038c60 2024-03-29T11:41:28.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:28.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:28.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:28.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b2c004eb0 con 0x7f8b44038c60 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079066384 0 0) 0x7f8b44036620 con 0x7f8b440d11e0 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b44036900 con 0x7f8b440d11e0 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22161458 0 0) 0x7f8b440363b0 con 0x7f8b440324b0 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:41:28.275 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b44036620 con 0x7f8b440324b0 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 299107242 0 0) 0x7f8b44036620 con 0x7f8b440324b0 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b440363b0 con 0x7f8b440324b0 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8b3c001020 con 0x7f8b440324b0 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): dump: 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:28.276 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.266+0000 7f8b3affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.267+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4145051843 0 0) 0x7f8b44036900 con 0x7f8b440d11e0 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.267+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b44036620 con 0x7f8b440d11e0 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.267+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8b4c04fdd0 con 0x7f8b440d11e0 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.267+0000 7f8b3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.267+0000 7f8b3affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.267+0000 7f8b3affd640 10 monclient(hunting): dump: 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:28.277 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2345705023 0 0) 0x7f8b44036620 con 0x7f8b440d11e0 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 >> v1:172.21.15.100:6790/0 conn(0x7f8b44038c60 legacy=0x7f8b440c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 >> v1:172.21.15.100:6789/0 conn(0x7f8b440324b0 legacy=0x7f8b4402d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:28.278 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 1 monclient: found mon.b 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b44035020 con 0x7f8b440d11e0 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 10 monclient: _finish_auth 0 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b4c04f400 con 0x7f8b440d11e0 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8b4c075710 con 0x7f8b440d11e0 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b3affd640 10 monclient: dump: 2024-03-29T11:41:28.279 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b4b59e640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b4b59e640 10 monclient: _renew_subs 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b4b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:41:28.280 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b4b59e640 1 -- 172.21.15.100:0/3435148383 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b440359a0 con 0x7f8b440d11e0 2024-03-29T11:41:28.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.268+0000 7f8b48d99640 4 set_mon_vals no callback set 2024-03-29T11:41:28.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.273+0000 7f8b4b59e640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:28.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.273+0000 7f8b4b59e640 10 monclient: _renew_subs 2024-03-29T11:41:28.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.273+0000 7f8b4b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:41:28.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.273+0000 7f8b4b59e640 1 -- 172.21.15.100:0/3435148383 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b440beac0 con 0x7f8b440d11e0 2024-03-29T11:41:28.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.273+0000 7f8b4b59e640 1 librados: init done 2024-03-29T11:41:28.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.274+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f8b4c067bf0 con 0x7f8b440d11e0 2024-03-29T11:41:28.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.274+0000 7f8b4b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:28.283 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.274+0000 7f8b4b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:28.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.274+0000 7f8b4b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:41:28.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.274+0000 7f8b4b59e640 1 -- 172.21.15.100:0/3435148383 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b440303c0 con 0x7f8b440d11e0 2024-03-29T11:41:28.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.275+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f8b4c0ff250 con 0x7f8b440d11e0 2024-03-29T11:41:28.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.275+0000 7f8b3affd640 10 client.4375.objecter ms_dispatch 0x7f8b4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:28.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.275+0000 7f8b3affd640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:28.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.275+0000 7f8b3affd640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:28.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.275+0000 7f8b4a59c640 10 monclient: get_auth_request con 0x7f8b2c06ddd0 auth_method 0 2024-03-29T11:41:28.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.275+0000 7f8b3affd640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T11:41:28.284 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.276+0000 7f8b3affd640 10 client.4375.objecter ms_handle_connect 0x7f8b2c06ddd0 2024-03-29T11:41:28.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.283+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b440303c0 con 0x7f8b440d11e0 2024-03-29T11:41:28.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.283+0000 7f8b3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:28.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.283+0000 7f8b3affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:28.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.617+0000 7f8b4b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:28.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.617+0000 7f8b4b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:28.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.617+0000 7f8b4b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:41:28.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.617+0000 7f8b4b59e640 1 -- 172.21.15.100:0/3435148383 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b440303c0 con 0x7f8b440d11e0 2024-03-29T11:41:28.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.619+0000 7f8b3affd640 1 -- 172.21.15.100:0/3435148383 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f8b440303c0 con 0x7f8b440d11e0 2024-03-29T11:41:28.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.619+0000 7f8b3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:28.623 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.619+0000 7f8b3affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:28.623 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:41:28.624 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:28.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.623+0000 7f8b4b59e640 10 librados: watch_flush enter 2024-03-29T11:41:28.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.623+0000 7f8b4b59e640 10 librados: watch_flush exit 2024-03-29T11:41:28.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.623+0000 7f8b4b59e640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T11:41:28.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.623+0000 7f8b4b59e640 10 client.4375.objecter successfully canceled tick 2024-03-29T11:41:28.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.624+0000 7f8b4b59e640 1 -- 172.21.15.100:0/3435148383 >> v1:172.21.15.100:6816/3220289657 conn(0x7f8b2c06ddd0 legacy=0x7f8b2c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:28.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.624+0000 7f8b4b59e640 10 monclient: shutdown 2024-03-29T11:41:28.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.624+0000 7f8b4b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:28.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.624+0000 7f8b4b59e640 1 -- 172.21.15.100:0/3435148383 >> v1:172.21.15.191:6789/0 conn(0x7f8b440d11e0 legacy=0x7f8b4416b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:28.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.624+0000 7f8b4b59e640 1 -- 172.21.15.100:0/3435148383 shutdown_connections 2024-03-29T11:41:28.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.624+0000 7f8b4b59e640 1 -- 172.21.15.100:0/3435148383 >> 172.21.15.100:0/3435148383 conn(0x7f8b44005c20 msgr2=0x7f8b44006000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:28.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.625+0000 7f8b4b59e640 1 -- 172.21.15.100:0/3435148383 shutdown_connections 2024-03-29T11:41:28.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.625+0000 7f8b4b59e640 1 -- 172.21.15.100:0/3435148383 wait complete. 2024-03-29T11:41:28.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:28.625+0000 7f8b4b59e640 1 librados: shutdown 2024-03-29T11:41:28.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:35.452 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:35.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.601+0000 7fdd713e2640 10 monclient: get_monmap_and_config 2024-03-29T11:41:35.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.601+0000 7fdd713e2640 10 monclient: build_initial_monmap 2024-03-29T11:41:35.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.601+0000 7fdd713e2640 10 monclient: monmap: 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:41:35.602299+0000 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:41:35.602299+0000 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:35.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.602+0000 7fdd713e2640 1 Processor -- start 2024-03-29T11:41:35.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.602+0000 7fdd713e2640 1 -- start start 2024-03-29T11:41:35.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.602+0000 7fdd713e2640 10 monclient: init 2024-03-29T11:41:35.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.603+0000 7fdd713e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:35.606 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.603+0000 7fdd713e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:35.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.603+0000 7fdd713e2640 10 monclient(hunting): picked mon.noname-b con 0x7fdd6c0b4100 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:35.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.603+0000 7fdd713e2640 10 monclient(hunting): picked mon.noname-a con 0x7fdd6c13ee90 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:35.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.603+0000 7fdd713e2640 10 monclient(hunting): picked mon.noname-c con 0x7fdd6c0af900 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:35.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.603+0000 7fdd713e2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c0afd10 con 0x7fdd6c13ee90 2024-03-29T11:41:35.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.603+0000 7fdd713e2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c0af6f0 con 0x7fdd6c0af900 2024-03-29T11:41:35.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.603+0000 7fdd713e2640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c0aac80 con 0x7fdd6c0b4100 2024-03-29T11:41:35.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.603+0000 7fdd713e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:35.607 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.603+0000 7fdd713e2640 10 monclient(hunting): authenticate will time out at 1215.798265s 2024-03-29T11:41:35.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.605+0000 7fdd6affd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fdd6c0b4100 0x7fdd6c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:45052/0 (socket says 172.21.15.100:45052) 2024-03-29T11:41:35.608 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.606+0000 7fdd6affd640 1 -- 172.21.15.100:0/3206630831 learned_addr learned my addr 172.21.15.100:0/3206630831 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:35.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.606+0000 7fdd69ffb640 1 -- 172.21.15.100:0/3206630831 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484856720 0 0) 0x7fdd6c0aac80 con 0x7fdd6c0b4100 2024-03-29T11:41:35.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.606+0000 7fdd69ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:35.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.606+0000 7fdd69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:35.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.606+0000 7fdd69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:35.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.606+0000 7fdd69ffb640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:41:35.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.606+0000 7fdd69ffb640 1 -- 172.21.15.100:0/3206630831 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd54003520 con 0x7fdd6c0b4100 2024-03-29T11:41:35.609 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.607+0000 7fdd69ffb640 1 -- 172.21.15.100:0/3206630831 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3273041229 0 0) 0x7fdd54003520 con 0x7fdd6c0b4100 2024-03-29T11:41:35.610 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.607+0000 7fdd69ffb640 1 -- 172.21.15.100:0/3206630831 >> v1:172.21.15.191:6789/0 conn(0x7fdd6c0af900 legacy=0x7fdd6c14be10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:35.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.608+0000 7fdd69ffb640 1 -- 172.21.15.100:0/3206630831 >> v1:172.21.15.100:6789/0 conn(0x7fdd6c13ee90 legacy=0x7fdd6c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:35.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:35.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 1 monclient: found mon.noname-b 2024-03-29T11:41:35.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:35.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:41:35.611 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 1 -- 172.21.15.100:0/3206630831 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c0aaf60 con 0x7fdd6c0b4100 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd713e2640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd713e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 1 -- 172.21.15.100:0/3206630831 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fdd5c001a10 con 0x7fdd6c0b4100 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 10 monclient: dump: 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:35.612 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd713e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 1 -- 172.21.15.100:0/3206630831 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd5c001be0 con 0x7fdd6c0b4100 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd713e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:35.613 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.609+0000 7fdd713e2640 4 set_mon_vals no callback set 2024-03-29T11:41:35.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.610+0000 7fdd713e2640 10 monclient: shutdown 2024-03-29T11:41:35.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.610+0000 7fdd713e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:35.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.610+0000 7fdd713e2640 1 -- 172.21.15.100:0/3206630831 >> v1:172.21.15.100:6790/0 conn(0x7fdd6c0b4100 legacy=0x7fdd6c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:35.614 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.610+0000 7fdd713e2640 1 -- 172.21.15.100:0/3206630831 shutdown_connections 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.610+0000 7fdd713e2640 1 -- 172.21.15.100:0/3206630831 >> 172.21.15.100:0/3206630831 conn(0x7fdd6c0b35e0 msgr2=0x7fdd6c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.610+0000 7fdd713e2640 1 -- 172.21.15.100:0/3206630831 shutdown_connections 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.610+0000 7fdd713e2640 1 -- 172.21.15.100:0/3206630831 wait complete. 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.611+0000 7fdd713e2640 10 monclient: build_initial_monmap 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.611+0000 7fdd713e2640 10 monclient: monmap: 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:41:35.615 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:41:35.616 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:35.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.611+0000 7fdd713e2640 1 librados: starting msgr at 2024-03-29T11:41:35.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.611+0000 7fdd713e2640 1 librados: starting objecter 2024-03-29T11:41:35.616 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.612+0000 7fdd713e2640 1 Processor -- start 2024-03-29T11:41:35.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.612+0000 7fdd713e2640 1 -- start start 2024-03-29T11:41:35.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.612+0000 7fdd713e2640 1 librados: setting wanted keys 2024-03-29T11:41:35.617 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.612+0000 7fdd713e2640 1 librados: calling monclient init 2024-03-29T11:41:35.618 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.612+0000 7fdd713e2640 10 monclient: init 2024-03-29T11:41:35.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.612+0000 7fdd713e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:35.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.612+0000 7fdd713e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:35.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.613+0000 7fdd713e2640 10 monclient(hunting): picked mon.noname-b con 0x7fdd6c0af900 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:35.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.613+0000 7fdd713e2640 10 monclient(hunting): picked mon.noname-a con 0x7fdd6c0b4100 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:35.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.613+0000 7fdd713e2640 10 monclient(hunting): picked mon.noname-c con 0x7fdd6c13ee90 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:35.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.613+0000 7fdd713e2640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c1df300 con 0x7fdd6c0b4100 2024-03-29T11:41:35.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.613+0000 7fdd713e2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c143560 con 0x7fdd6c0af900 2024-03-29T11:41:35.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.613+0000 7fdd713e2640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c1dee20 con 0x7fdd6c13ee90 2024-03-29T11:41:35.619 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.613+0000 7fdd713e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:35.620 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.613+0000 7fdd713e2640 10 monclient(hunting): authenticate will time out at 1215.808139s 2024-03-29T11:41:35.621 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.617+0000 7fdd6b7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fdd6c13ee90 0x7fdd6c142e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:48560/0 (socket says 172.21.15.100:48560) 2024-03-29T11:41:35.621 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.617+0000 7fdd6b7fe640 1 -- 172.21.15.100:0/1946959356 learned_addr learned my addr 172.21.15.100:0/1946959356 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:35.621 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 client.?.objecter ms_handle_connect 0x7fdd6c13ee90 2024-03-29T11:41:35.621 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:35.621 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 client.?.objecter ms_handle_connect 0x7fdd6c0af900 2024-03-29T11:41:35.621 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:35.621 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225079941 0 0) 0x7fdd6c143560 con 0x7fdd6c0af900 2024-03-29T11:41:35.621 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:35.621 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:35.622 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:35.622 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:41:35.622 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd58004eb0 con 0x7fdd6c0af900 2024-03-29T11:41:35.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1190929528 0 0) 0x7fdd58004eb0 con 0x7fdd6c0af900 2024-03-29T11:41:35.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd6c143560 con 0x7fdd6c0af900 2024-03-29T11:41:35.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fdd5c004cc0 con 0x7fdd6c0af900 2024-03-29T11:41:35.626 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 monclient(hunting): dump: 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:35.627 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 client.?.objecter ms_handle_connect 0x7fdd6c0b4100 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.619+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018546192 0 0) 0x7fdd6c1dee20 con 0x7fdd6c13ee90 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd58004eb0 con 0x7fdd6c13ee90 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334932157 0 0) 0x7fdd6c1df300 con 0x7fdd6c0b4100 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:35.628 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd6c1dee20 con 0x7fdd6c0b4100 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2525442753 0 0) 0x7fdd6c143560 con 0x7fdd6c0af900 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 >> v1:172.21.15.100:6790/0 conn(0x7fdd6c13ee90 legacy=0x7fdd6c142e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 >> v1:172.21.15.100:6789/0 conn(0x7fdd6c0b4100 legacy=0x7fdd6c142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 1 monclient: found mon.b 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c1defb0 con 0x7fdd6c0af900 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:35.629 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd713e2640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd713e2640 10 monclient: _renew_subs 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd713e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.620+0000 7fdd713e2640 1 -- 172.21.15.100:0/1946959356 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c140c50 con 0x7fdd6c0af900 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.621+0000 7fdd713e2640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.621+0000 7fdd713e2640 10 monclient: _renew_subs 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.621+0000 7fdd713e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.621+0000 7fdd713e2640 1 -- 172.21.15.100:0/1946959356 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c141180 con 0x7fdd6c0af900 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.621+0000 7fdd713e2640 1 librados: init done 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.622+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd5c001e90 con 0x7fdd6c0af900 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.622+0000 7fdd537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.622+0000 7fdd697fa640 4 set_mon_vals no callback set 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.622+0000 7fdd713e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:35.630 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.622+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fdd5c005270 con 0x7fdd6c0af900 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.622+0000 7fdd537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.622+0000 7fdd537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.622+0000 7fdd537fe640 10 monclient: dump: 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:35.631 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.623+0000 7fdd713e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:35.632 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.623+0000 7fdd713e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:41:35.632 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.623+0000 7fdd713e2640 1 -- 172.21.15.100:0/1946959356 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4c005080 con 0x7fdd6c0af900 2024-03-29T11:41:35.632 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.624+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fdd5c01b240 con 0x7fdd6c0af900 2024-03-29T11:41:35.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.625+0000 7fdd6a7fc640 10 monclient: get_auth_request con 0x7fdd5806de60 auth_method 0 2024-03-29T11:41:35.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.625+0000 7fdd537fe640 10 client.4378.objecter ms_handle_connect 0x7fdd5806de60 2024-03-29T11:41:35.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.625+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fdd5c085070 con 0x7fdd6c0af900 2024-03-29T11:41:35.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.626+0000 7fdd537fe640 10 client.4378.objecter ms_dispatch 0x7fdd6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:35.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.626+0000 7fdd537fe640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:35.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.626+0000 7fdd537fe640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:35.633 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.626+0000 7fdd537fe640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-29T11:41:35.634 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.632+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd5c050950 con 0x7fdd6c0af900 2024-03-29T11:41:35.635 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.632+0000 7fdd537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:35.635 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.632+0000 7fdd537fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:35.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.961+0000 7fdd713e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:35.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.961+0000 7fdd713e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:35.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.961+0000 7fdd713e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:41:35.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.961+0000 7fdd713e2640 1 -- 172.21.15.100:0/1946959356 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd4c005330 con 0x7fdd6c0af900 2024-03-29T11:41:35.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.961+0000 7fdd537fe640 1 -- 172.21.15.100:0/1946959356 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fdd5c052880 con 0x7fdd6c0af900 2024-03-29T11:41:35.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.961+0000 7fdd537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:35.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.961+0000 7fdd537fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:35.965 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:41:35.965 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:35.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 10 librados: watch_flush enter 2024-03-29T11:41:35.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 10 librados: watch_flush exit 2024-03-29T11:41:35.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-29T11:41:35.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 10 client.4378.objecter successfully canceled tick 2024-03-29T11:41:35.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 1 -- 172.21.15.100:0/1946959356 >> v1:172.21.15.100:6816/3220289657 conn(0x7fdd5806de60 legacy=0x7fdd58070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:35.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 10 monclient: shutdown 2024-03-29T11:41:35.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:35.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 1 -- 172.21.15.100:0/1946959356 >> v1:172.21.15.191:6789/0 conn(0x7fdd6c0af900 legacy=0x7fdd6c13fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:35.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 1 -- 172.21.15.100:0/1946959356 shutdown_connections 2024-03-29T11:41:35.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 1 -- 172.21.15.100:0/1946959356 >> 172.21.15.100:0/1946959356 conn(0x7fdd6c0b35e0 msgr2=0x7fdd6c0aceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:35.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 1 -- 172.21.15.100:0/1946959356 shutdown_connections 2024-03-29T11:41:35.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.966+0000 7fdd713e2640 1 -- 172.21.15.100:0/1946959356 wait complete. 2024-03-29T11:41:35.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:35.967+0000 7fdd713e2640 1 librados: shutdown 2024-03-29T11:41:35.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:42.792 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:42.947 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.944+0000 7f38c9c4a640 10 monclient: get_monmap_and_config 2024-03-29T11:41:42.947 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.944+0000 7f38c9c4a640 10 monclient: build_initial_monmap 2024-03-29T11:41:42.947 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.944+0000 7f38c9c4a640 10 monclient: monmap: 2024-03-29T11:41:42.947 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:42.947 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:42.947 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:41:42.945640+0000 2024-03-29T11:41:42.947 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:41:42.945640+0000 2024-03-29T11:41:42.947 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:42.948 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:42.948 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:42.948 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:41:42.948 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:41:42.948 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:42.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.945+0000 7f38c9c4a640 1 Processor -- start 2024-03-29T11:41:42.948 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.945+0000 7f38c9c4a640 1 -- start start 2024-03-29T11:41:42.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.945+0000 7f38c9c4a640 10 monclient: init 2024-03-29T11:41:42.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.946+0000 7f38c9c4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:42.949 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.946+0000 7f38c9c4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:42.952 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.946+0000 7f38c9c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f38c41457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:42.952 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.946+0000 7f38c9c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f38c413f650 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:42.952 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.946+0000 7f38c9c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f38c40a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:42.952 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.946+0000 7f38c9c4a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c40aaef0 con 0x7f38c41457f0 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.946+0000 7f38c9c4a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c4145bd0 con 0x7f38c413f650 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.946+0000 7f38c9c4a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c4145eb0 con 0x7f38c40a9d70 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.946+0000 7f38c9c4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.946+0000 7f38c9c4a640 10 monclient(hunting): authenticate will time out at 1223.141909s 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.947+0000 7f38c2ffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f38c413f650 0x7f38c40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:49344/0 (socket says 172.21.15.100:49344) 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.947+0000 7f38c2ffd640 1 -- 172.21.15.100:0/1784644123 learned_addr learned my addr 172.21.15.100:0/1784644123 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.947+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643252850 0 0) 0x7f38c4145bd0 con 0x7f38c413f650 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.947+0000 7f38c27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.947+0000 7f38c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.947+0000 7f38c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.947+0000 7f38c27fc640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.947+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b0003520 con 0x7f38c413f650 2024-03-29T11:41:42.953 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872238522 0 0) 0x7f38c4145eb0 con 0x7f38c40a9d70 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c4145bd0 con 0x7f38c40a9d70 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3657570416 0 0) 0x7f38b0003520 con 0x7f38c413f650 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 >> v1:172.21.15.100:6790/0 conn(0x7f38c40a9d70 legacy=0x7f38c414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 >> v1:172.21.15.100:6789/0 conn(0x7f38c41457f0 legacy=0x7f38c413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 1 monclient: found mon.noname-c 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:41:42.954 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c4146340 con 0x7f38c413f650 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.948+0000 7f38c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f38ac001690 con 0x7f38c413f650 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c27fc640 10 monclient: dump: 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:42.955 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:42.956 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:42.956 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:42.956 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:42.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c9c4a640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:41:42.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c9c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:42.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38ac0017f0 con 0x7f38c413f650 2024-03-29T11:41:42.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:42.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c27fc640 1 -- 172.21.15.100:0/1784644123 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f38ac001c40 con 0x7f38c413f650 2024-03-29T11:41:42.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:42.956 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c27fc640 10 monclient: dump: 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c9c4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:42.957 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.949+0000 7f38c9c4a640 4 set_mon_vals no callback set 2024-03-29T11:41:42.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.950+0000 7f38c9c4a640 10 monclient: shutdown 2024-03-29T11:41:42.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.950+0000 7f38c9c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:42.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.950+0000 7f38c9c4a640 1 -- 172.21.15.100:0/1784644123 >> v1:172.21.15.191:6789/0 conn(0x7f38c413f650 legacy=0x7f38c40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:42.961 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.950+0000 7f38c9c4a640 1 -- 172.21.15.100:0/1784644123 shutdown_connections 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.950+0000 7f38c9c4a640 1 -- 172.21.15.100:0/1784644123 >> 172.21.15.100:0/1784644123 conn(0x7f38c4066a00 msgr2=0x7f38c4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.951+0000 7f38c9c4a640 1 -- 172.21.15.100:0/1784644123 shutdown_connections 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.951+0000 7f38c9c4a640 1 -- 172.21.15.100:0/1784644123 wait complete. 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.952+0000 7f38c9c4a640 10 monclient: build_initial_monmap 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.952+0000 7f38c9c4a640 10 monclient: monmap: 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:42.962 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.952+0000 7f38c9c4a640 1 librados: starting msgr at 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.952+0000 7f38c9c4a640 1 librados: starting objecter 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.952+0000 7f38c9c4a640 1 Processor -- start 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.952+0000 7f38c9c4a640 1 -- start start 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.952+0000 7f38c9c4a640 1 librados: setting wanted keys 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.952+0000 7f38c9c4a640 1 librados: calling monclient init 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.952+0000 7f38c9c4a640 10 monclient: init 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c9c4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c9c4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c9c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f38c40a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c9c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f38c413f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c2ffd640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f38c413f650 0x7f38c41da630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:59652/0 (socket says 172.21.15.100:59652) 2024-03-29T11:41:42.963 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c2ffd640 1 -- 172.21.15.100:0/2258458875 learned_addr learned my addr 172.21.15.100:0/2258458875 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c9c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f38c41457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c40b1880 con 0x7f38c413f650 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c40add50 con 0x7f38c41457f0 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c40ae1e0 con 0x7f38c40a9d70 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c9c4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38c9c4a640 10 monclient(hunting): authenticate will time out at 1223.148873s 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.953+0000 7f38bbfff640 10 client.?.objecter ms_handle_connect 0x7f38c413f650 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 client.?.objecter ms_handle_connect 0x7f38c40a9d70 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 client.?.objecter ms_handle_connect 0x7f38c41457f0 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:42.964 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347318604 0 0) 0x7f38c40b1880 con 0x7f38c413f650 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a4004eb0 con 0x7f38c413f650 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693168032 0 0) 0x7f38c40ae1e0 con 0x7f38c40a9d70 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c40b1880 con 0x7f38c40a9d70 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2638851601 0 0) 0x7f38a4004eb0 con 0x7f38c413f650 2024-03-29T11:41:42.965 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38c40ae1e0 con 0x7f38c413f650 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.954+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f38ac001b00 con 0x7f38c413f650 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient(hunting): dump: 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:42.966 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:42.967 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:42.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:42.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 407417885 0 0) 0x7f38c40b1880 con 0x7f38c40a9d70 2024-03-29T11:41:42.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38a4004eb0 con 0x7f38c40a9d70 2024-03-29T11:41:42.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f38b4001740 con 0x7f38c40a9d70 2024-03-29T11:41:42.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:42.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:42.967 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient(hunting): dump: 2024-03-29T11:41:42.967 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:42.967 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3129692526 0 0) 0x7f38a4004eb0 con 0x7f38c40a9d70 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 >> v1:172.21.15.191:6789/0 conn(0x7f38c41457f0 legacy=0x7f38c41e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 >> v1:172.21.15.100:6789/0 conn(0x7f38c413f650 legacy=0x7f38c41da630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:42.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 1 monclient: found mon.c 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c40ad750 con 0x7f38c40a9d70 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38c9c4a640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38c9c4a640 10 monclient: _renew_subs 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38c9c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c40ada00 con 0x7f38c40a9d70 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38c9c4a640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38c9c4a640 10 monclient: _renew_subs 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38c9c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:42.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c40b1550 con 0x7f38c40a9d70 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.955+0000 7f38c9c4a640 1 librados: init done 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.956+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38b4001c10 con 0x7f38c40a9d70 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.956+0000 7f38bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.956+0000 7f38c1ffb640 4 set_mon_vals no callback set 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.957+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f38b40037b0 con 0x7f38c40a9d70 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.957+0000 7f38bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.957+0000 7f38bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.957+0000 7f38bbfff640 10 monclient: dump: 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:42.970 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:42.971 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:42.971 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:42.971 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:42.971 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:42.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.957+0000 7f38c9c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:42.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.957+0000 7f38c9c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:42.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.957+0000 7f38c9c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:42.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.957+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a8005080 con 0x7f38c40a9d70 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.959+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f38b4019780 con 0x7f38c40a9d70 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.960+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f38b4083600 con 0x7f38c40a9d70 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.960+0000 7f38bbfff640 10 client.4445.objecter ms_dispatch 0x7f38c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.960+0000 7f38bbfff640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.960+0000 7f38bbfff640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.960+0000 7f38bbfff640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.960+0000 7f38c2ffd640 10 monclient: get_auth_request con 0x7f38a406dea0 auth_method 0 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.961+0000 7f38bbfff640 10 client.4445.objecter ms_handle_connect 0x7f38a406dea0 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.969+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38b404ed10 con 0x7f38c40a9d70 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.969+0000 7f38bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:42.972 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:42.969+0000 7f38bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:43.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.360+0000 7f38c9c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:43.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.360+0000 7f38c9c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:43.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.360+0000 7f38c9c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:43.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.360+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38a8005330 con 0x7f38c40a9d70 2024-03-29T11:41:43.365 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.362+0000 7f38bbfff640 1 -- 172.21.15.100:0/2258458875 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f38b4050e10 con 0x7f38c40a9d70 2024-03-29T11:41:43.365 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.362+0000 7f38bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:43.365 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.362+0000 7f38bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:43.365 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:41:43.366 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:43.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.366+0000 7f38c9c4a640 10 librados: watch_flush enter 2024-03-29T11:41:43.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.366+0000 7f38c9c4a640 10 librados: watch_flush exit 2024-03-29T11:41:43.369 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.366+0000 7f38c9c4a640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:41:43.369 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.366+0000 7f38c9c4a640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:41:43.369 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.366+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 >> v1:172.21.15.100:6816/3220289657 conn(0x7f38a406dea0 legacy=0x7f38a4070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:43.369 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.366+0000 7f38c9c4a640 10 monclient: shutdown 2024-03-29T11:41:43.369 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.366+0000 7f38c9c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:43.369 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.366+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 >> v1:172.21.15.100:6790/0 conn(0x7f38c40a9d70 legacy=0x7f38c40b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:43.369 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.367+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 shutdown_connections 2024-03-29T11:41:43.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.367+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 >> 172.21.15.100:0/2258458875 conn(0x7f38c4066a00 msgr2=0x7f38c4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:43.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.367+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 shutdown_connections 2024-03-29T11:41:43.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.367+0000 7f38c9c4a640 1 -- 172.21.15.100:0/2258458875 wait complete. 2024-03-29T11:41:43.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:43.367+0000 7f38c9c4a640 1 librados: shutdown 2024-03-29T11:41:43.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:50.191 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:50.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.333+0000 7f286f548640 10 monclient: get_monmap_and_config 2024-03-29T11:41:50.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.333+0000 7f286f548640 10 monclient: build_initial_monmap 2024-03-29T11:41:50.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.333+0000 7f286f548640 10 monclient: monmap: 2024-03-29T11:41:50.336 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:50.336 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:50.337 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:41:50.334895+0000 2024-03-29T11:41:50.337 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:41:50.334895+0000 2024-03-29T11:41:50.337 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:50.337 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:50.337 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:50.337 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:41:50.337 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:41:50.337 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:50.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.334+0000 7f286f548640 1 Processor -- start 2024-03-29T11:41:50.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.334+0000 7f286f548640 1 -- start start 2024-03-29T11:41:50.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.335+0000 7f286f548640 10 monclient: init 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.335+0000 7f286f548640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.335+0000 7f286f548640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.335+0000 7f286f548640 10 monclient(hunting): picked mon.noname-c con 0x7f28681457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.336+0000 7f286f548640 10 monclient(hunting): picked mon.noname-a con 0x7f286813f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.336+0000 7f286dae5640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f28680a9d70 0x7f286814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:59820/0 (socket says 172.21.15.100:59820) 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.336+0000 7f286dae5640 1 -- 172.21.15.100:0/3291525640 learned_addr learned my addr 172.21.15.100:0/3291525640 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.336+0000 7f286f548640 10 monclient(hunting): picked mon.noname-b con 0x7f28680a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.336+0000 7f286f548640 1 -- 172.21.15.100:0/3291525640 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28680aaef0 con 0x7f286813f650 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.336+0000 7f286f548640 1 -- 172.21.15.100:0/3291525640 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2868145bd0 con 0x7f28681457f0 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.336+0000 7f286f548640 1 -- 172.21.15.100:0/3291525640 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2868145eb0 con 0x7f28680a9d70 2024-03-29T11:41:50.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.336+0000 7f286f548640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:50.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.336+0000 7f286f548640 10 monclient(hunting): authenticate will time out at 1230.532018s 2024-03-29T11:41:50.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.337+0000 7f285ffff640 1 -- 172.21.15.100:0/3291525640 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827001540 0 0) 0x7f2868145eb0 con 0x7f28680a9d70 2024-03-29T11:41:50.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.337+0000 7f285ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:50.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.337+0000 7f285ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:50.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.337+0000 7f285ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:50.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.337+0000 7f285ffff640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:41:50.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.337+0000 7f285ffff640 1 -- 172.21.15.100:0/3291525640 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2854003520 con 0x7f28680a9d70 2024-03-29T11:41:50.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.337+0000 7f285ffff640 1 -- 172.21.15.100:0/3291525640 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2243502210 0 0) 0x7f2854003520 con 0x7f28680a9d70 2024-03-29T11:41:50.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.337+0000 7f285ffff640 1 -- 172.21.15.100:0/3291525640 >> v1:172.21.15.191:6789/0 conn(0x7f28681457f0 legacy=0x7f286813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:50.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.339+0000 7f285ffff640 1 -- 172.21.15.100:0/3291525640 >> v1:172.21.15.100:6789/0 conn(0x7f286813f650 legacy=0x7f28680b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:50.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.341+0000 7f285ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:50.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.341+0000 7f285ffff640 1 monclient: found mon.noname-b 2024-03-29T11:41:50.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.341+0000 7f285ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:50.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.341+0000 7f285ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.341+0000 7f285ffff640 1 -- 172.21.15.100:0/3291525640 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868146340 con 0x7f28680a9d70 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.341+0000 7f285ffff640 10 monclient: _finish_auth 0 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 1 -- 172.21.15.100:0/3291525640 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2860001a10 con 0x7f28680a9d70 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f286f548640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f286f548640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 10 monclient: dump: 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:50.347 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f286f548640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 1 -- 172.21.15.100:0/3291525640 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2860001ca0 con 0x7f28680a9d70 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f286f548640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f286f548640 4 set_mon_vals no callback set 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 1 -- 172.21.15.100:0/3291525640 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2860002030 con 0x7f28680a9d70 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.342+0000 7f285ffff640 10 monclient: dump: 2024-03-29T11:41:50.348 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.343+0000 7f286f548640 10 monclient: shutdown 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.343+0000 7f286f548640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.343+0000 7f286f548640 1 -- 172.21.15.100:0/3291525640 >> v1:172.21.15.100:6790/0 conn(0x7f28680a9d70 legacy=0x7f286814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.344+0000 7f286f548640 1 -- 172.21.15.100:0/3291525640 shutdown_connections 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.344+0000 7f286f548640 1 -- 172.21.15.100:0/3291525640 >> 172.21.15.100:0/3291525640 conn(0x7f2868066a00 msgr2=0x7f2868066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.344+0000 7f286f548640 1 -- 172.21.15.100:0/3291525640 shutdown_connections 2024-03-29T11:41:50.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.344+0000 7f286f548640 1 -- 172.21.15.100:0/3291525640 wait complete. 2024-03-29T11:41:50.351 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.346+0000 7f286f548640 10 monclient: build_initial_monmap 2024-03-29T11:41:50.351 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.346+0000 7f286f548640 10 monclient: monmap: 2024-03-29T11:41:50.351 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:50.351 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:50.351 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:41:50.351 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:41:50.351 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:50.351 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:50.351 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.346+0000 7f286f548640 1 librados: starting msgr at 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.346+0000 7f286f548640 1 librados: starting objecter 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.346+0000 7f286f548640 1 Processor -- start 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.347+0000 7f286f548640 1 -- start start 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.347+0000 7f286f548640 1 librados: setting wanted keys 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.347+0000 7f286f548640 1 librados: calling monclient init 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.347+0000 7f286f548640 10 monclient: init 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.348+0000 7f286f548640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:50.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.348+0000 7f286f548640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:50.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.351+0000 7f286f548640 10 monclient(hunting): picked mon.noname-b con 0x7f28680a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:50.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.351+0000 7f286d2e4640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f28680a9d70 0x7f286813ad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:46937/0 (socket says 172.21.15.100:46937) 2024-03-29T11:41:50.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.351+0000 7f286d2e4640 1 -- 172.21.15.100:0/2607479593 learned_addr learned my addr 172.21.15.100:0/2607479593 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:50.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.352+0000 7f286f548640 10 monclient(hunting): picked mon.noname-a con 0x7f286813f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:50.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.352+0000 7f286f548640 10 monclient(hunting): picked mon.noname-c con 0x7f28681457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:50.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.352+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286813a670 con 0x7f286813f650 2024-03-29T11:41:50.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.352+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286813a950 con 0x7f28680a9d70 2024-03-29T11:41:50.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.352+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2868138e00 con 0x7f28681457f0 2024-03-29T11:41:50.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.352+0000 7f286f548640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:50.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.352+0000 7f286f548640 10 monclient(hunting): authenticate will time out at 1230.547614s 2024-03-29T11:41:50.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.353+0000 7f285dffb640 10 client.?.objecter ms_handle_connect 0x7f28680a9d70 2024-03-29T11:41:50.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.353+0000 7f285dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:50.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.353+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700226504 0 0) 0x7f286813a950 con 0x7f28680a9d70 2024-03-29T11:41:50.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.353+0000 7f285dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:50.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.353+0000 7f285dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:50.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.353+0000 7f285dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:50.363 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.353+0000 7f285dffb640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.353+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f284c004eb0 con 0x7f28680a9d70 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.354+0000 7f285dffb640 10 client.?.objecter ms_handle_connect 0x7f28681457f0 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.354+0000 7f285dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.354+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519502321 0 0) 0x7f2868138e00 con 0x7f28681457f0 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.354+0000 7f285dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.354+0000 7f285dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.354+0000 7f285dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.354+0000 7f285dffb640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f286813a950 con 0x7f28681457f0 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 582568920 0 0) 0x7f286813a950 con 0x7f28681457f0 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2868138e00 con 0x7f28681457f0 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f28600018d0 con 0x7f28681457f0 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:50.364 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:50.365 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 10 monclient(hunting): dump: 2024-03-29T11:41:50.365 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:50.365 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:50.365 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:50.365 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:50.365 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:50.365 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:50.365 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2622655049 0 0) 0x7f2868138e00 con 0x7f28681457f0 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 >> v1:172.21.15.191:6789/0 conn(0x7f28680a9d70 legacy=0x7f286813ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 >> v1:172.21.15.100:6789/0 conn(0x7f286813f650 legacy=0x7f28681350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 1 monclient: found mon.c 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868138a90 con 0x7f28681457f0 2024-03-29T11:41:50.366 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f285dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f286f548640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f286f548640 10 monclient: _renew_subs 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f286f548640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.355+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28681396c0 con 0x7f28681457f0 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2860004f70 con 0x7f28681457f0 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f285dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f286f548640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f286f548640 10 monclient: _renew_subs 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f286f548640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2868139500 con 0x7f28681457f0 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2860005280 con 0x7f28681457f0 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f286f548640 1 librados: init done 2024-03-29T11:41:50.367 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f285dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:50.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f285dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:50.368 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.356+0000 7f285dffb640 10 monclient: dump: 2024-03-29T11:41:50.368 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:50.368 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:50.368 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:50.368 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:50.368 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:50.368 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:50.369 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:50.369 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:50.369 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:50.369 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:50.369 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.358+0000 7f286f548640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:50.369 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.358+0000 7f286f548640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:50.369 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.358+0000 7f286f548640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:50.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.358+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2868075f10 con 0x7f28681457f0 2024-03-29T11:41:50.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.358+0000 7f285effd640 4 set_mon_vals no callback set 2024-03-29T11:41:50.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.359+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f286001b250 con 0x7f28681457f0 2024-03-29T11:41:50.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.360+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f2860084d30 con 0x7f28681457f0 2024-03-29T11:41:50.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.360+0000 7f285dffb640 10 client.4451.objecter ms_dispatch 0x7f2868000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:50.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.360+0000 7f285dffb640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:50.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.360+0000 7f285dffb640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:50.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.360+0000 7f285dffb640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:41:50.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.360+0000 7f286d2e4640 10 monclient: get_auth_request con 0x7f284c076490 auth_method 0 2024-03-29T11:41:50.370 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.360+0000 7f285dffb640 10 client.4451.objecter ms_handle_connect 0x7f284c076490 2024-03-29T11:41:50.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.367+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2860050610 con 0x7f28681457f0 2024-03-29T11:41:50.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.367+0000 7f285dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:50.371 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.367+0000 7f285dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:50.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.701+0000 7f286f548640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:50.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.701+0000 7f286f548640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:50.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.701+0000 7f286f548640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:50.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.701+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2868139500 con 0x7f28681457f0 2024-03-29T11:41:50.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.702+0000 7f285dffb640 1 -- 172.21.15.100:0/2607479593 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f2868139500 con 0x7f28681457f0 2024-03-29T11:41:50.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.702+0000 7f285dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:50.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.702+0000 7f285dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:50.705 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:41:50.705 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:50.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.706+0000 7f286f548640 10 librados: watch_flush enter 2024-03-29T11:41:50.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.706+0000 7f286f548640 10 librados: watch_flush exit 2024-03-29T11:41:50.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.706+0000 7f286f548640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:41:50.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.706+0000 7f286f548640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:41:50.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.706+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 >> v1:172.21.15.100:6816/3220289657 conn(0x7f284c076490 legacy=0x7f284c078950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:50.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.706+0000 7f286f548640 10 monclient: shutdown 2024-03-29T11:41:50.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.706+0000 7f286f548640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:50.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.706+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 >> v1:172.21.15.100:6790/0 conn(0x7f28681457f0 legacy=0x7f28681e2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:50.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.706+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 shutdown_connections 2024-03-29T11:41:50.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.706+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 >> 172.21.15.100:0/2607479593 conn(0x7f2868066a00 msgr2=0x7f28680b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:50.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.707+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 shutdown_connections 2024-03-29T11:41:50.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.707+0000 7f286f548640 1 -- 172.21.15.100:0/2607479593 wait complete. 2024-03-29T11:41:50.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:50.707+0000 7f286f548640 1 librados: shutdown 2024-03-29T11:41:50.724 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:57.531 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:57.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.683+0000 7f5f22f46640 10 monclient: get_monmap_and_config 2024-03-29T11:41:57.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.683+0000 7f5f22f46640 10 monclient: build_initial_monmap 2024-03-29T11:41:57.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.683+0000 7f5f22f46640 10 monclient: monmap: 2024-03-29T11:41:57.686 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:57.686 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:57.686 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:41:57.684639+0000 2024-03-29T11:41:57.687 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:41:57.684639+0000 2024-03-29T11:41:57.687 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:57.687 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:57.687 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:57.687 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:41:57.687 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:41:57.687 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:57.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.684+0000 7f5f22f46640 1 Processor -- start 2024-03-29T11:41:57.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.684+0000 7f5f22f46640 1 -- start start 2024-03-29T11:41:57.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.684+0000 7f5f22f46640 10 monclient: init 2024-03-29T11:41:57.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.685+0000 7f5f22f46640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.685+0000 7f5f22f46640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.686+0000 7f5f22f46640 10 monclient(hunting): picked mon.noname-a con 0x7f5f1c1457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.686+0000 7f5f22f46640 10 monclient(hunting): picked mon.noname-b con 0x7f5f1c13f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.686+0000 7f5f20ce2640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f5f1c1457f0 0x7f5f1c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:53836/0 (socket says 172.21.15.100:53836) 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.686+0000 7f5f20ce2640 1 -- 172.21.15.100:0/4016985158 learned_addr learned my addr 172.21.15.100:0/4016985158 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.686+0000 7f5f22f46640 10 monclient(hunting): picked mon.noname-c con 0x7f5f1c0a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.686+0000 7f5f22f46640 1 -- 172.21.15.100:0/4016985158 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c0aaef0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.686+0000 7f5f22f46640 1 -- 172.21.15.100:0/4016985158 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c145bd0 con 0x7f5f1c0a9d70 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.686+0000 7f5f22f46640 1 -- 172.21.15.100:0/4016985158 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c145eb0 con 0x7f5f1c13f650 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.686+0000 7f5f22f46640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:57.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.686+0000 7f5f22f46640 10 monclient(hunting): authenticate will time out at 1237.881477s 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562959893 0 0) 0x7f5f1c0aaef0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f08003520 con 0x7f5f1c1457f0 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439572879 0 0) 0x7f5f1c145eb0 con 0x7f5f1c13f650 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:41:57.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f1c0aaef0 con 0x7f5f1c13f650 2024-03-29T11:41:57.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571164536 0 0) 0x7f5f1c145bd0 con 0x7f5f1c0a9d70 2024-03-29T11:41:57.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:57.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:57.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:57.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:41:57.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f1c145eb0 con 0x7f5f1c0a9d70 2024-03-29T11:41:57.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187244340 0 0) 0x7f5f08003520 con 0x7f5f1c1457f0 2024-03-29T11:41:57.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 >> v1:172.21.15.100:6790/0 conn(0x7f5f1c13f650 legacy=0x7f5f1c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:57.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 >> v1:172.21.15.191:6789/0 conn(0x7f5f1c0a9d70 legacy=0x7f5f1c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c146340 con 0x7f5f1c1457f0 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.689+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5f0c001a10 con 0x7f5f1c1457f0 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.690+0000 7f5f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.690+0000 7f5f1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.690+0000 7f5f1b7fe640 10 monclient: dump: 2024-03-29T11:41:57.695 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.690+0000 7f5f22f46640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:41:57.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.690+0000 7f5f22f46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:57.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.691+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f0c001c30 con 0x7f5f1c1457f0 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.691+0000 7f5f1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.691+0000 7f5f1b7fe640 1 -- 172.21.15.100:0/4016985158 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5f0c001fc0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.691+0000 7f5f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.691+0000 7f5f1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.691+0000 7f5f1b7fe640 10 monclient: dump: 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:57.699 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.692+0000 7f5f22f46640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.692+0000 7f5f22f46640 4 set_mon_vals no callback set 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.693+0000 7f5f22f46640 10 monclient: shutdown 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.693+0000 7f5f22f46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.693+0000 7f5f22f46640 1 -- 172.21.15.100:0/4016985158 >> v1:172.21.15.100:6789/0 conn(0x7f5f1c1457f0 legacy=0x7f5f1c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.693+0000 7f5f22f46640 1 -- 172.21.15.100:0/4016985158 shutdown_connections 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.693+0000 7f5f22f46640 1 -- 172.21.15.100:0/4016985158 >> 172.21.15.100:0/4016985158 conn(0x7f5f1c066a00 msgr2=0x7f5f1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.693+0000 7f5f22f46640 1 -- 172.21.15.100:0/4016985158 shutdown_connections 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.693+0000 7f5f22f46640 1 -- 172.21.15.100:0/4016985158 wait complete. 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.694+0000 7f5f22f46640 10 monclient: build_initial_monmap 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.694+0000 7f5f22f46640 10 monclient: monmap: 2024-03-29T11:41:57.700 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.695+0000 7f5f22f46640 1 librados: starting msgr at 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.695+0000 7f5f22f46640 1 librados: starting objecter 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.695+0000 7f5f22f46640 1 Processor -- start 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.695+0000 7f5f22f46640 1 -- start start 2024-03-29T11:41:57.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.695+0000 7f5f22f46640 1 librados: setting wanted keys 2024-03-29T11:41:57.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.695+0000 7f5f22f46640 1 librados: calling monclient init 2024-03-29T11:41:57.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.695+0000 7f5f22f46640 10 monclient: init 2024-03-29T11:41:57.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f22f46640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:57.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f22f46640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:57.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f22f46640 10 monclient(hunting): picked mon.noname-b con 0x7f5f1c0a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:41:57.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f22f46640 10 monclient(hunting): picked mon.noname-a con 0x7f5f1c13f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:41:57.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f22f46640 10 monclient(hunting): picked mon.noname-c con 0x7f5f1c1457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:41:57.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f22f46640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c0ac0f0 con 0x7f5f1c13f650 2024-03-29T11:41:57.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f22f46640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c0ac3d0 con 0x7f5f1c0a9d70 2024-03-29T11:41:57.709 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f22f46640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c0ac6b0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f22f46640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f22f46640 10 monclient(hunting): authenticate will time out at 1237.891672s 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f214e3640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5f1c1457f0 0x7f5f1c1373b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:44482/0 (socket says 172.21.15.100:44482) 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f214e3640 1 -- 172.21.15.100:0/1113285337 learned_addr learned my addr 172.21.15.100:0/1113285337 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f197fa640 10 client.?.objecter ms_handle_connect 0x7f5f1c1457f0 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.696+0000 7f5f197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101856758 0 0) 0x7f5f1c0ac6b0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5efc004eb0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.710 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3231949103 0 0) 0x7f5efc004eb0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f1c0ac6b0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5f14001a40 con 0x7f5f1c1457f0 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 10 monclient(hunting): dump: 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:57.711 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.697+0000 7f5f197fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.698+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1513871126 0 0) 0x7f5f1c0ac6b0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.698+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 >> v1:172.21.15.191:6789/0 conn(0x7f5f1c0a9d70 legacy=0x7f5f1c142560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 >> v1:172.21.15.100:6789/0 conn(0x7f5f1c13f650 legacy=0x7f5f1c136ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 10 monclient: _finish_hunting 0 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 1 monclient: found mon.c 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c137ac0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 10 monclient: _finish_auth 0 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 10 monclient: _check_auth_tickets 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:57.712 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f22f46640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f22f46640 10 monclient: _renew_subs 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f22f46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f22f46640 1 -- 172.21.15.100:0/1113285337 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1c13fda0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f22f46640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f22f46640 10 monclient: _renew_subs 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f22f46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f22f46640 1 -- 172.21.15.100:0/1113285337 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1c141c10 con 0x7f5f1c1457f0 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f22f46640 1 librados: init done 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f14001d20 con 0x7f5f1c1457f0 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f1a7fc640 4 set_mon_vals no callback set 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5f14003a60 con 0x7f5f1c1457f0 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:57.713 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.699+0000 7f5f197fa640 10 monclient: dump: 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.701+0000 7f5f22f46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.701+0000 7f5f22f46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:57.714 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.701+0000 7f5f22f46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:57.715 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.701+0000 7f5f22f46640 1 -- 172.21.15.100:0/1113285337 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f00005080 con 0x7f5f1c1457f0 2024-03-29T11:41:57.715 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.702+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f5f14003c40 con 0x7f5f1c1457f0 2024-03-29T11:41:57.715 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.703+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f5f140548b0 con 0x7f5f1c1457f0 2024-03-29T11:41:57.715 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.703+0000 7f5f197fa640 10 client.4457.objecter ms_dispatch 0x7f5f1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:57.715 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.703+0000 7f5f197fa640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:57.715 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.703+0000 7f5f197fa640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:57.715 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.703+0000 7f5f197fa640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T11:41:57.715 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.703+0000 7f5f20ce2640 10 monclient: get_auth_request con 0x7f5efc06dc80 auth_method 0 2024-03-29T11:41:57.715 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.703+0000 7f5f197fa640 10 client.4457.objecter ms_handle_connect 0x7f5efc06dc80 2024-03-29T11:41:57.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.714+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f14050b50 con 0x7f5f1c1457f0 2024-03-29T11:41:57.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.714+0000 7f5f197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:57.717 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:57.714+0000 7f5f197fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:58.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.078+0000 7f5f22f46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:58.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.078+0000 7f5f22f46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:58.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.078+0000 7f5f22f46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:41:58.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.078+0000 7f5f22f46640 1 -- 172.21.15.100:0/1113285337 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f00005330 con 0x7f5f1c1457f0 2024-03-29T11:41:58.082 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.079+0000 7f5f197fa640 1 -- 172.21.15.100:0/1113285337 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f5f14050b50 con 0x7f5f1c1457f0 2024-03-29T11:41:58.082 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.079+0000 7f5f197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:58.082 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.079+0000 7f5f197fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:58.082 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:41:58.082 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:58.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.082+0000 7f5f22f46640 10 librados: watch_flush enter 2024-03-29T11:41:58.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.083+0000 7f5f22f46640 10 librados: watch_flush exit 2024-03-29T11:41:58.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.083+0000 7f5f22f46640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T11:41:58.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.083+0000 7f5f22f46640 10 client.4457.objecter successfully canceled tick 2024-03-29T11:41:58.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.083+0000 7f5f22f46640 1 -- 172.21.15.100:0/1113285337 >> v1:172.21.15.100:6816/3220289657 conn(0x7f5efc06dc80 legacy=0x7f5efc070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:58.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.083+0000 7f5f22f46640 10 monclient: shutdown 2024-03-29T11:41:58.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.083+0000 7f5f22f46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:58.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.083+0000 7f5f22f46640 1 -- 172.21.15.100:0/1113285337 >> v1:172.21.15.100:6790/0 conn(0x7f5f1c1457f0 legacy=0x7f5f1c1373b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:58.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.084+0000 7f5f22f46640 1 -- 172.21.15.100:0/1113285337 shutdown_connections 2024-03-29T11:41:58.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.084+0000 7f5f22f46640 1 -- 172.21.15.100:0/1113285337 >> 172.21.15.100:0/1113285337 conn(0x7f5f1c066a00 msgr2=0x7f5f1c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:58.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.084+0000 7f5f22f46640 1 -- 172.21.15.100:0/1113285337 shutdown_connections 2024-03-29T11:41:58.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.085+0000 7f5f22f46640 1 -- 172.21.15.100:0/1113285337 wait complete. 2024-03-29T11:41:58.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:41:58.085+0000 7f5f22f46640 1 librados: shutdown 2024-03-29T11:41:58.105 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:04.912 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:05.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.064+0000 7f02cee51640 10 monclient: get_monmap_and_config 2024-03-29T11:42:05.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.064+0000 7f02cee51640 10 monclient: build_initial_monmap 2024-03-29T11:42:05.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.064+0000 7f02cee51640 10 monclient: monmap: 2024-03-29T11:42:05.067 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:05.067 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:05.067 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:42:05.065367+0000 2024-03-29T11:42:05.067 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:42:05.065367+0000 2024-03-29T11:42:05.067 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:05.068 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:05.068 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:05.068 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:42:05.068 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:42:05.068 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:05.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.065+0000 7f02cee51640 1 Processor -- start 2024-03-29T11:42:05.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.066+0000 7f02cee51640 1 -- start start 2024-03-29T11:42:05.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.066+0000 7f02cee51640 10 monclient: init 2024-03-29T11:42:05.069 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cee51640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:05.076 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cee51640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:05.076 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cee51640 10 monclient(hunting): picked mon.noname-b con 0x7f02c0006e30 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cee51640 10 monclient(hunting): picked mon.noname-c con 0x7f02c0035e00 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cee51640 10 monclient(hunting): picked mon.noname-a con 0x7f02c00c9670 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cee51640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c0035970 con 0x7f02c00c9670 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cee51640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c0036280 con 0x7f02c0035e00 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cee51640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c0035640 con 0x7f02c0006e30 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cee51640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cee51640 10 monclient(hunting): authenticate will time out at 1245.262204s 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cde4f640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f02c0006e30 0x7f02c00cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:51488/0 (socket says 172.21.15.100:51488) 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.067+0000 7f02cde4f640 1 -- 172.21.15.100:0/4140691122 learned_addr learned my addr 172.21.15.100:0/4140691122 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.070+0000 7f02cce4d640 1 -- 172.21.15.100:0/4140691122 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645712395 0 0) 0x7f02c0035640 con 0x7f02c0006e30 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.070+0000 7f02cce4d640 10 monclient(hunting): _negotiate 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.070+0000 7f02cce4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:05.077 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.070+0000 7f02cce4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.070+0000 7f02cce4d640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.070+0000 7f02cce4d640 1 -- 172.21.15.100:0/4140691122 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b4003520 con 0x7f02c0006e30 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.070+0000 7f02cce4d640 1 -- 172.21.15.100:0/4140691122 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3440204159 0 0) 0x7f02b4003520 con 0x7f02c0006e30 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.070+0000 7f02cce4d640 1 -- 172.21.15.100:0/4140691122 >> v1:172.21.15.191:6789/0 conn(0x7f02c0035e00 legacy=0x7f02c00c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.070+0000 7f02cce4d640 1 -- 172.21.15.100:0/4140691122 >> v1:172.21.15.100:6789/0 conn(0x7f02c00c9670 legacy=0x7f02c00d7560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: _finish_hunting 0 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 1 monclient: found mon.noname-b 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 1 -- 172.21.15.100:0/4140691122 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c0034e20 con 0x7f02c0006e30 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: _finish_auth 0 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: _check_auth_tickets 2024-03-29T11:42:05.078 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cee51640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cee51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 1 -- 172.21.15.100:0/4140691122 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f02c4001690 con 0x7f02c0006e30 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: dump: 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:05.079 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cee51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 1 -- 172.21.15.100:0/4140691122 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02c4001920 con 0x7f02c0006e30 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 1 -- 172.21.15.100:0/4140691122 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f02c4001cb0 con 0x7f02c0006e30 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cce4d640 10 monclient: dump: 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:05.080 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cee51640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.071+0000 7f02cee51640 4 set_mon_vals no callback set 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.073+0000 7f02cee51640 10 monclient: shutdown 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.073+0000 7f02cee51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.073+0000 7f02cee51640 1 -- 172.21.15.100:0/4140691122 >> v1:172.21.15.100:6790/0 conn(0x7f02c0006e30 legacy=0x7f02c00cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:05.081 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.074+0000 7f02cee51640 1 -- 172.21.15.100:0/4140691122 shutdown_connections 2024-03-29T11:42:05.082 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.074+0000 7f02cee51640 1 -- 172.21.15.100:0/4140691122 >> 172.21.15.100:0/4140691122 conn(0x7f02c0005c00 msgr2=0x7f02c0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:05.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.075+0000 7f02cee51640 1 -- 172.21.15.100:0/4140691122 shutdown_connections 2024-03-29T11:42:05.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.075+0000 7f02cee51640 1 -- 172.21.15.100:0/4140691122 wait complete. 2024-03-29T11:42:05.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.076+0000 7f02cee51640 10 monclient: build_initial_monmap 2024-03-29T11:42:05.083 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.076+0000 7f02cee51640 10 monclient: monmap: 2024-03-29T11:42:05.083 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:05.083 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:05.083 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:42:05.083 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.076+0000 7f02cee51640 1 librados: starting msgr at 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.076+0000 7f02cee51640 1 librados: starting objecter 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.076+0000 7f02cee51640 1 Processor -- start 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.079+0000 7f02cee51640 1 -- start start 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.079+0000 7f02cee51640 1 librados: setting wanted keys 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.079+0000 7f02cee51640 1 librados: calling monclient init 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.079+0000 7f02cee51640 10 monclient: init 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.079+0000 7f02cee51640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:05.084 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.079+0000 7f02cee51640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:05.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.083+0000 7f02cee51640 10 monclient(hunting): picked mon.noname-a con 0x7f02c0006e30 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:05.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.083+0000 7f02cee51640 10 monclient(hunting): picked mon.noname-c con 0x7f02c0035e00 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:05.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.083+0000 7f02cee51640 10 monclient(hunting): picked mon.noname-b con 0x7f02c00c9670 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:05.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.083+0000 7f02cee51640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c00321f0 con 0x7f02c0006e30 2024-03-29T11:42:05.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.083+0000 7f02cee51640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c00cdc50 con 0x7f02c00c9670 2024-03-29T11:42:05.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.083+0000 7f02cee51640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c00cdec0 con 0x7f02c0035e00 2024-03-29T11:42:05.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.083+0000 7f02cee51640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:05.086 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.083+0000 7f02cee51640 10 monclient(hunting): authenticate will time out at 1245.278927s 2024-03-29T11:42:05.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.084+0000 7f02cd64e640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f02c0035e00 0x7f02c00335b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:53098/0 (socket says 172.21.15.100:53098) 2024-03-29T11:42:05.087 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.084+0000 7f02cd64e640 1 -- 172.21.15.100:0/2227509830 learned_addr learned my addr 172.21.15.100:0/2227509830 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:05.088 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 client.?.objecter ms_handle_connect 0x7f02c0035e00 2024-03-29T11:42:05.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:05.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028049297 0 0) 0x7f02c00cdec0 con 0x7f02c0035e00 2024-03-29T11:42:05.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:05.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:05.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:05.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:42:05.089 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ac004eb0 con 0x7f02c0035e00 2024-03-29T11:42:05.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 client.?.objecter ms_handle_connect 0x7f02c0006e30 2024-03-29T11:42:05.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:05.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258683209 0 0) 0x7f02c0034e20 con 0x7f02c0006e30 2024-03-29T11:42:05.090 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.086+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c00cdec0 con 0x7f02c0006e30 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3148099884 0 0) 0x7f02c00cdec0 con 0x7f02c0006e30 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02c0034e20 con 0x7f02c0006e30 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f02c40013b0 con 0x7f02c0006e30 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:05.091 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3892490285 0 0) 0x7f02ac004eb0 con 0x7f02c0035e00 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02c00cdec0 con 0x7f02c0035e00 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f02c804fdd0 con 0x7f02c0035e00 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:05.092 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2861626204 0 0) 0x7f02c0034e20 con 0x7f02c0006e30 2024-03-29T11:42:05.093 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 >> v1:172.21.15.100:6790/0 conn(0x7f02c0035e00 legacy=0x7f02c00335b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 >> v1:172.21.15.191:6789/0 conn(0x7f02c00c9670 legacy=0x7f02c01741e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 monclient: found mon.a 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c0033dc0 con 0x7f02c0006e30 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02cee51640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02cee51640 10 monclient: _renew_subs 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02cee51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:42:05.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02cee51640 1 -- 172.21.15.100:0/2227509830 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c00cff30 con 0x7f02c0006e30 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02cee51640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02cee51640 10 monclient: _renew_subs 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02cee51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02cee51640 1 -- 172.21.15.100:0/2227509830 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c0032a30 con 0x7f02c0006e30 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02cee51640 1 librados: init done 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02c4001690 con 0x7f02c0006e30 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.087+0000 7f02be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.088+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f02c4001510 con 0x7f02c0006e30 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.088+0000 7f02be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.088+0000 7f02be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.088+0000 7f02be7fc640 10 monclient: dump: 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:05.095 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:05.096 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:05.096 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:05.096 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:05.096 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:05.096 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:05.096 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:05.096 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:05.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.088+0000 7f02bffff640 4 set_mon_vals no callback set 2024-03-29T11:42:05.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.091+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f02c401ac00 con 0x7f02c0006e30 2024-03-29T11:42:05.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.091+0000 7f02cee51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:05.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.091+0000 7f02cee51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:05.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.091+0000 7f02cee51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:42:05.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.091+0000 7f02cee51640 1 -- 172.21.15.100:0/2227509830 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b0005080 con 0x7f02c0006e30 2024-03-29T11:42:05.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.092+0000 7f02cd64e640 10 monclient: get_auth_request con 0x7f02ac06ddb0 auth_method 0 2024-03-29T11:42:05.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.093+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f02c4084a40 con 0x7f02c0006e30 2024-03-29T11:42:05.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.093+0000 7f02be7fc640 10 client.4407.objecter ms_dispatch 0x7f02c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:05.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.093+0000 7f02be7fc640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:05.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.093+0000 7f02be7fc640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:05.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.093+0000 7f02be7fc640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T11:42:05.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.093+0000 7f02be7fc640 10 client.4407.objecter ms_handle_connect 0x7f02ac06ddb0 2024-03-29T11:42:05.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.101+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02c4050320 con 0x7f02c0006e30 2024-03-29T11:42:05.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.101+0000 7f02be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:05.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.101+0000 7f02be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:05.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.417+0000 7f02cee51640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:05.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.417+0000 7f02cee51640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:05.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.417+0000 7f02cee51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:42:05.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.417+0000 7f02cee51640 1 -- 172.21.15.100:0/2227509830 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02b0005330 con 0x7f02c0006e30 2024-03-29T11:42:05.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.418+0000 7f02be7fc640 1 -- 172.21.15.100:0/2227509830 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f02c4052250 con 0x7f02c0006e30 2024-03-29T11:42:05.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.418+0000 7f02be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:05.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.418+0000 7f02be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:05.422 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:42:05.422 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:05.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.421+0000 7f02cee51640 10 librados: watch_flush enter 2024-03-29T11:42:05.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.422+0000 7f02cee51640 10 librados: watch_flush exit 2024-03-29T11:42:05.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.422+0000 7f02cee51640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T11:42:05.425 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.422+0000 7f02cee51640 10 client.4407.objecter successfully canceled tick 2024-03-29T11:42:05.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.423+0000 7f02cee51640 1 -- 172.21.15.100:0/2227509830 >> v1:172.21.15.100:6816/3220289657 conn(0x7f02ac06ddb0 legacy=0x7f02ac070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:05.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.423+0000 7f02cee51640 10 monclient: shutdown 2024-03-29T11:42:05.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.423+0000 7f02cee51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:05.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.423+0000 7f02cee51640 1 -- 172.21.15.100:0/2227509830 >> v1:172.21.15.100:6789/0 conn(0x7f02c0006e30 legacy=0x7f02c00cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:05.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.424+0000 7f02cee51640 1 -- 172.21.15.100:0/2227509830 shutdown_connections 2024-03-29T11:42:05.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.424+0000 7f02cee51640 1 -- 172.21.15.100:0/2227509830 >> 172.21.15.100:0/2227509830 conn(0x7f02c0005c00 msgr2=0x7f02c003ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:05.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.425+0000 7f02cee51640 1 -- 172.21.15.100:0/2227509830 shutdown_connections 2024-03-29T11:42:05.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.425+0000 7f02cee51640 1 -- 172.21.15.100:0/2227509830 wait complete. 2024-03-29T11:42:05.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:05.425+0000 7f02cee51640 1 librados: shutdown 2024-03-29T11:42:05.446 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:12.252 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:12.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.403+0000 7f2e88d99640 10 monclient: get_monmap_and_config 2024-03-29T11:42:12.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.403+0000 7f2e88d99640 10 monclient: build_initial_monmap 2024-03-29T11:42:12.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.404+0000 7f2e88d99640 10 monclient: monmap: 2024-03-29T11:42:12.407 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:12.407 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:12.407 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:42:12.405004+0000 2024-03-29T11:42:12.407 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:42:12.405004+0000 2024-03-29T11:42:12.407 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:12.407 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:12.407 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:12.408 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:42:12.408 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:42:12.408 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:12.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.407+0000 7f2e88d99640 1 Processor -- start 2024-03-29T11:42:12.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.407+0000 7f2e88d99640 1 -- start start 2024-03-29T11:42:12.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.407+0000 7f2e88d99640 10 monclient: init 2024-03-29T11:42:12.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.408+0000 7f2e88d99640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:12.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.408+0000 7f2e88d99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:12.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.408+0000 7f2e88d99640 10 monclient(hunting): picked mon.noname-c con 0x7f2e7c006890 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:12.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.408+0000 7f2e88d99640 10 monclient(hunting): picked mon.noname-b con 0x7f2e7c006c70 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:12.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.408+0000 7f2e88d99640 10 monclient(hunting): picked mon.noname-a con 0x7f2e7c02b4c0 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:12.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.408+0000 7f2e88d99640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c02c830 con 0x7f2e7c02b4c0 2024-03-29T11:42:12.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.408+0000 7f2e88d99640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c02afa0 con 0x7f2e7c006890 2024-03-29T11:42:12.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.408+0000 7f2e88d99640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c02aba0 con 0x7f2e7c006c70 2024-03-29T11:42:12.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.408+0000 7f2e88d99640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:12.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.408+0000 7f2e88d99640 10 monclient(hunting): authenticate will time out at 1252.603691s 2024-03-29T11:42:12.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.409+0000 7f2e82ffd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f2e7c006c70 0x7f2e7c0c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:39638/0 (socket says 172.21.15.100:39638) 2024-03-29T11:42:12.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.409+0000 7f2e82ffd640 1 -- 172.21.15.100:0/293005958 learned_addr learned my addr 172.21.15.100:0/293005958 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:12.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.412+0000 7f2e827fc640 1 -- 172.21.15.100:0/293005958 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214832530 0 0) 0x7f2e7c02aba0 con 0x7f2e7c006c70 2024-03-29T11:42:12.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.413+0000 7f2e827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.413+0000 7f2e827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.413+0000 7f2e827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.413+0000 7f2e827fc640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:42:12.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.413+0000 7f2e827fc640 1 -- 172.21.15.100:0/293005958 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e74003520 con 0x7f2e7c006c70 2024-03-29T11:42:12.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.413+0000 7f2e827fc640 1 -- 172.21.15.100:0/293005958 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2490002468 0 0) 0x7f2e74003520 con 0x7f2e7c006c70 2024-03-29T11:42:12.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.413+0000 7f2e827fc640 1 -- 172.21.15.100:0/293005958 >> v1:172.21.15.191:6789/0 conn(0x7f2e7c006890 legacy=0x7f2e7c0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.413+0000 7f2e827fc640 1 -- 172.21.15.100:0/293005958 >> v1:172.21.15.100:6789/0 conn(0x7f2e7c02b4c0 legacy=0x7f2e7c0cf0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:12.419 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:12.419 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 1 monclient: found mon.noname-b 2024-03-29T11:42:12.419 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:12.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:42:12.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 1 -- 172.21.15.100:0/293005958 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c02ad30 con 0x7f2e7c006c70 2024-03-29T11:42:12.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:12.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:12.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:12.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 1 -- 172.21.15.100:0/293005958 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2e8404f730 con 0x7f2e7c006c70 2024-03-29T11:42:12.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:12.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e827fc640 10 monclient: dump: 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e88d99640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:42:12.421 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.414+0000 7f2e88d99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:12.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.415+0000 7f2e827fc640 1 -- 172.21.15.100:0/293005958 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e84075f10 con 0x7f2e7c006c70 2024-03-29T11:42:12.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.415+0000 7f2e827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:12.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.415+0000 7f2e827fc640 1 -- 172.21.15.100:0/293005958 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2e840758b0 con 0x7f2e7c006c70 2024-03-29T11:42:12.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.415+0000 7f2e827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.415+0000 7f2e827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:12.424 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.415+0000 7f2e827fc640 10 monclient: dump: 2024-03-29T11:42:12.424 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.416+0000 7f2e88d99640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.416+0000 7f2e88d99640 4 set_mon_vals no callback set 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.417+0000 7f2e88d99640 10 monclient: shutdown 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.417+0000 7f2e88d99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:12.425 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.417+0000 7f2e88d99640 1 -- 172.21.15.100:0/293005958 >> v1:172.21.15.100:6790/0 conn(0x7f2e7c006c70 legacy=0x7f2e7c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.417+0000 7f2e88d99640 1 -- 172.21.15.100:0/293005958 shutdown_connections 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.417+0000 7f2e88d99640 1 -- 172.21.15.100:0/293005958 >> 172.21.15.100:0/293005958 conn(0x7f2e7c0058f0 msgr2=0x7f2e7c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.417+0000 7f2e88d99640 1 -- 172.21.15.100:0/293005958 shutdown_connections 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.417+0000 7f2e88d99640 1 -- 172.21.15.100:0/293005958 wait complete. 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.418+0000 7f2e88d99640 10 monclient: build_initial_monmap 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.418+0000 7f2e88d99640 10 monclient: monmap: 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:12.426 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.418+0000 7f2e88d99640 1 librados: starting msgr at 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.418+0000 7f2e88d99640 1 librados: starting objecter 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.419+0000 7f2e88d99640 1 Processor -- start 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.419+0000 7f2e88d99640 1 -- start start 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.419+0000 7f2e88d99640 1 librados: setting wanted keys 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.419+0000 7f2e88d99640 1 librados: calling monclient init 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.419+0000 7f2e88d99640 10 monclient: init 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.420+0000 7f2e88d99640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.420+0000 7f2e88d99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.420+0000 7f2e88d99640 10 monclient(hunting): picked mon.noname-a con 0x7f2e7c006890 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:12.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.420+0000 7f2e88d99640 10 monclient(hunting): picked mon.noname-b con 0x7f2e7c02b4c0 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:12.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.420+0000 7f2e88d99640 10 monclient(hunting): picked mon.noname-c con 0x7f2e7c02d1a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:12.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.420+0000 7f2e88d99640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c02e390 con 0x7f2e7c006890 2024-03-29T11:42:12.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.420+0000 7f2e88d99640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c02e670 con 0x7f2e7c02b4c0 2024-03-29T11:42:12.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.420+0000 7f2e88d99640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c02e950 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.420+0000 7f2e88d99640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:12.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.420+0000 7f2e88d99640 10 monclient(hunting): authenticate will time out at 1252.615310s 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.423+0000 7f2e83fff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f2e7c02d1a0 0x7f2e7c030e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:40216/0 (socket says 172.21.15.100:40216) 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.423+0000 7f2e83fff640 1 -- 172.21.15.100:0/1693170025 learned_addr learned my addr 172.21.15.100:0/1693170025 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.423+0000 7f2e6ffff640 10 client.?.objecter ms_handle_connect 0x7f2e7c02d1a0 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.423+0000 7f2e6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.424+0000 7f2e6ffff640 10 client.?.objecter ms_handle_connect 0x7f2e7c006890 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.424+0000 7f2e6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.424+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288325813 0 0) 0x7f2e7c02e950 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.424+0000 7f2e6ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.424+0000 7f2e6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.424+0000 7f2e6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.424+0000 7f2e6ffff640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:42:12.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.424+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e68004eb0 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4037961407 0 0) 0x7f2e68004eb0 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e7c02e950 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585671077 0 0) 0x7f2e7c02e390 con 0x7f2e7c006890 2024-03-29T11:42:12.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:12.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:12.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:12.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:42:12.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e68004eb0 con 0x7f2e7c006890 2024-03-29T11:42:12.434 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2e700018e0 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.425+0000 7f2e6ffff640 10 monclient(hunting): dump: 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:12.435 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2896243873 0 0) 0x7f2e7c02e950 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 >> v1:172.21.15.191:6789/0 conn(0x7f2e7c02b4c0 legacy=0x7f2e7c0d2330 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 >> v1:172.21.15.100:6789/0 conn(0x7f2e7c006890 legacy=0x7f2e7c0d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 1 monclient: found mon.c 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c0d44a0 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 10 monclient: _finish_auth 0 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:12.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e88d99640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e88d99640 10 monclient: _renew_subs 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e88d99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e88d99640 1 -- 172.21.15.100:0/1693170025 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c0d4750 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e88d99640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e88d99640 10 monclient: _renew_subs 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e88d99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e88d99640 1 -- 172.21.15.100:0/1693170025 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c031780 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.426+0000 7f2e88d99640 1 librados: init done 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.427+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e70001ab0 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.427+0000 7f2e6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.427+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2e700038a0 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.427+0000 7f2e6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:12.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.427+0000 7f2e6ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:12.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.427+0000 7f2e6ffff640 10 monclient: dump: 2024-03-29T11:42:12.438 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:12.438 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:12.438 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:12.438 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:12.438 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:12.438 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:12.438 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:12.438 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:12.439 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:12.439 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:12.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.428+0000 7f2e817fa640 4 set_mon_vals no callback set 2024-03-29T11:42:12.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.430+0000 7f2e88d99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.430+0000 7f2e88d99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.430+0000 7f2e88d99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:12.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.430+0000 7f2e88d99640 1 -- 172.21.15.100:0/1693170025 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e64005080 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.440 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.434+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f2e70003a00 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.435+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f2e700849c0 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.435+0000 7f2e6ffff640 10 client.4469.objecter ms_dispatch 0x7f2e84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:12.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.435+0000 7f2e6ffff640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:12.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.435+0000 7f2e6ffff640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:12.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.435+0000 7f2e6ffff640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T11:42:12.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.436+0000 7f2e837fe640 10 monclient: get_auth_request con 0x7f2e6806dec0 auth_method 0 2024-03-29T11:42:12.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.436+0000 7f2e6ffff640 10 client.4469.objecter ms_handle_connect 0x7f2e6806dec0 2024-03-29T11:42:12.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.441+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e700500d0 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.441+0000 7f2e6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:12.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.441+0000 7f2e6ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:12.776 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.772+0000 7f2e88d99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:12.776 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.772+0000 7f2e88d99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:12.776 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.772+0000 7f2e88d99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:12.776 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.772+0000 7f2e88d99640 1 -- 172.21.15.100:0/1693170025 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e64005330 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.775+0000 7f2e6ffff640 1 -- 172.21.15.100:0/1693170025 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f2e700521d0 con 0x7f2e7c02d1a0 2024-03-29T11:42:12.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.775+0000 7f2e6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:12.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.775+0000 7f2e6ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:12.779 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:42:12.779 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:12.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.779+0000 7f2e88d99640 10 librados: watch_flush enter 2024-03-29T11:42:12.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.779+0000 7f2e88d99640 10 librados: watch_flush exit 2024-03-29T11:42:12.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.779+0000 7f2e88d99640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T11:42:12.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.779+0000 7f2e88d99640 10 client.4469.objecter successfully canceled tick 2024-03-29T11:42:12.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.780+0000 7f2e88d99640 1 -- 172.21.15.100:0/1693170025 >> v1:172.21.15.100:6816/3220289657 conn(0x7f2e6806dec0 legacy=0x7f2e68070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.780+0000 7f2e88d99640 10 monclient: shutdown 2024-03-29T11:42:12.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.780+0000 7f2e88d99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:12.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.780+0000 7f2e88d99640 1 -- 172.21.15.100:0/1693170025 >> v1:172.21.15.100:6790/0 conn(0x7f2e7c02d1a0 legacy=0x7f2e7c030e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:12.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.780+0000 7f2e88d99640 1 -- 172.21.15.100:0/1693170025 shutdown_connections 2024-03-29T11:42:12.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.780+0000 7f2e88d99640 1 -- 172.21.15.100:0/1693170025 >> 172.21.15.100:0/1693170025 conn(0x7f2e7c0058f0 msgr2=0x7f2e7c038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:12.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.781+0000 7f2e88d99640 1 -- 172.21.15.100:0/1693170025 shutdown_connections 2024-03-29T11:42:12.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.781+0000 7f2e88d99640 1 -- 172.21.15.100:0/1693170025 wait complete. 2024-03-29T11:42:12.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:12.781+0000 7f2e88d99640 1 librados: shutdown 2024-03-29T11:42:12.801 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:19.608 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:19.742 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.738+0000 7f5eed8ae640 10 monclient: get_monmap_and_config 2024-03-29T11:42:19.742 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.738+0000 7f5eed8ae640 10 monclient: build_initial_monmap 2024-03-29T11:42:19.742 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.738+0000 7f5eed8ae640 10 monclient: monmap: 2024-03-29T11:42:19.742 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:42:19.739922+0000 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:42:19.739922+0000 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.739+0000 7f5eed8ae640 1 Processor -- start 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.739+0000 7f5eed8ae640 1 -- start start 2024-03-29T11:42:19.743 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.739+0000 7f5eed8ae640 10 monclient: init 2024-03-29T11:42:19.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.740+0000 7f5eed8ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:19.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.740+0000 7f5eed8ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:19.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.740+0000 7f5eed8ae640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee80b0770 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:19.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.740+0000 7f5eed8ae640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee80ad850 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:19.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.740+0000 7f5eed8ae640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee813c1b0 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:19.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.740+0000 7f5eed8ae640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee80ac8e0 con 0x7f5ee80b0770 2024-03-29T11:42:19.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.740+0000 7f5ee67fc640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5ee80ad850 0x7f5ee80aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:58706/0 (socket says 172.21.15.100:58706) 2024-03-29T11:42:19.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.740+0000 7f5ee67fc640 1 -- 172.21.15.100:0/2687138830 learned_addr learned my addr 172.21.15.100:0/2687138830 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:19.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.740+0000 7f5eed8ae640 1 -- 172.21.15.100:0/2687138830 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee80ad330 con 0x7f5ee813c1b0 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.740+0000 7f5eed8ae640 1 -- 172.21.15.100:0/2687138830 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee80acfe0 con 0x7f5ee80ad850 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.741+0000 7f5eed8ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.741+0000 7f5eed8ae640 10 monclient(hunting): authenticate will time out at 1259.936181s 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.741+0000 7f5ee5ffb640 1 -- 172.21.15.100:0/2687138830 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097742967 0 0) 0x7f5ee80acfe0 con 0x7f5ee80ad850 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.741+0000 7f5ee5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.741+0000 7f5ee5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.741+0000 7f5ee5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.741+0000 7f5ee5ffb640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.741+0000 7f5ee5ffb640 1 -- 172.21.15.100:0/2687138830 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed0003520 con 0x7f5ee80ad850 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.742+0000 7f5ee5ffb640 1 -- 172.21.15.100:0/2687138830 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 813318461 0 0) 0x7f5ed0003520 con 0x7f5ee80ad850 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.742+0000 7f5ee5ffb640 1 -- 172.21.15.100:0/2687138830 >> v1:172.21.15.191:6789/0 conn(0x7f5ee813c1b0 legacy=0x7f5ee814a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:19.747 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.742+0000 7f5ee5ffb640 1 -- 172.21.15.100:0/2687138830 >> v1:172.21.15.100:6789/0 conn(0x7f5ee80b0770 legacy=0x7f5ee8072070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:19.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:19.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:42:19.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:19.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:42:19.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 1 -- 172.21.15.100:0/2687138830 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee80ab1f0 con 0x7f5ee80ad850 2024-03-29T11:42:19.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:19.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:19.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:19.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 1 -- 172.21.15.100:0/2687138830 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5edc001670 con 0x7f5ee80ad850 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: dump: 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:19.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5eed8ae640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5eed8ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 1 -- 172.21.15.100:0/2687138830 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5edc0018d0 con 0x7f5ee80ad850 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 1 -- 172.21.15.100:0/2687138830 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5edc001d20 con 0x7f5ee80ad850 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5ee5ffb640 10 monclient: dump: 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:19.752 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5eed8ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.744+0000 7f5eed8ae640 4 set_mon_vals no callback set 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.745+0000 7f5eed8ae640 10 monclient: shutdown 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.745+0000 7f5eed8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.745+0000 7f5eed8ae640 1 -- 172.21.15.100:0/2687138830 >> v1:172.21.15.100:6790/0 conn(0x7f5ee80ad850 legacy=0x7f5ee80aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.745+0000 7f5eed8ae640 1 -- 172.21.15.100:0/2687138830 shutdown_connections 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.745+0000 7f5eed8ae640 1 -- 172.21.15.100:0/2687138830 >> 172.21.15.100:0/2687138830 conn(0x7f5ee806b660 msgr2=0x7f5ee806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.746+0000 7f5eed8ae640 1 -- 172.21.15.100:0/2687138830 shutdown_connections 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.746+0000 7f5eed8ae640 1 -- 172.21.15.100:0/2687138830 wait complete. 2024-03-29T11:42:19.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.747+0000 7f5eed8ae640 10 monclient: build_initial_monmap 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.747+0000 7f5eed8ae640 10 monclient: monmap: 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:42:19.754 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.748+0000 7f5eed8ae640 1 librados: starting msgr at 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.748+0000 7f5eed8ae640 1 librados: starting objecter 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.748+0000 7f5eed8ae640 1 Processor -- start 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.748+0000 7f5eed8ae640 1 -- start start 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.749+0000 7f5eed8ae640 1 librados: setting wanted keys 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.749+0000 7f5eed8ae640 1 librados: calling monclient init 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.749+0000 7f5eed8ae640 10 monclient: init 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.750+0000 7f5eed8ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.750+0000 7f5eed8ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.750+0000 7f5eed8ae640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee80b0770 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.750+0000 7f5ee6ffd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5ee80b0770 0x7f5ee81dbb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:60266/0 (socket says 172.21.15.100:60266) 2024-03-29T11:42:19.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5eed8ae640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee813c1b0 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ee6ffd640 1 -- 172.21.15.100:0/339557220 learned_addr learned my addr 172.21.15.100:0/339557220 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5eed8ae640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee806dda0 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee8070470 con 0x7f5ee806dda0 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee81db5e0 con 0x7f5ee813c1b0 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee81dc210 con 0x7f5ee80b0770 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5eed8ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 10 client.?.objecter ms_handle_connect 0x7f5ee80b0770 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5eed8ae640 10 monclient(hunting): authenticate will time out at 1259.946290s 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462764083 0 0) 0x7f5ee81dc210 con 0x7f5ee80b0770 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:19.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed4004eb0 con 0x7f5ee80b0770 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3471476011 0 0) 0x7f5ed4004eb0 con 0x7f5ee80b0770 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ee81dc210 con 0x7f5ee80b0770 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 10 client.?.objecter ms_handle_connect 0x7f5ee813c1b0 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5ed8001960 con 0x7f5ee80b0770 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.751+0000 7f5ecffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 10 monclient(hunting): dump: 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:19.761 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518948362 0 0) 0x7f5ee81db5e0 con 0x7f5ee813c1b0 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:19.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed4004eb0 con 0x7f5ee813c1b0 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3390005342 0 0) 0x7f5ee81dc210 con 0x7f5ee80b0770 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 >> v1:172.21.15.191:6789/0 conn(0x7f5ee813c1b0 legacy=0x7f5ee806eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.752+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 >> v1:172.21.15.100:6789/0 conn(0x7f5ee806dda0 legacy=0x7f5ee81e5cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.754+0000 7f5ecffff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.754+0000 7f5ecffff640 1 monclient: found mon.c 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.754+0000 7f5ecffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.754+0000 7f5ecffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.754+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee81dc3a0 con 0x7f5ee80b0770 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5ecffff640 10 monclient: _finish_auth 0 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5ecffff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5ecffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:19.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5eed8ae640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:42:19.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5eed8ae640 10 monclient: _renew_subs 2024-03-29T11:42:19.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5eed8ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:19.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee806f250 con 0x7f5ee80b0770 2024-03-29T11:42:19.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5eed8ae640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:19.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5eed8ae640 10 monclient: _renew_subs 2024-03-29T11:42:19.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5eed8ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:19.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee81e75f0 con 0x7f5ee80b0770 2024-03-29T11:42:19.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.755+0000 7f5eed8ae640 1 librados: init done 2024-03-29T11:42:19.765 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.756+0000 7f5eed8ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.756+0000 7f5eed8ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.756+0000 7f5eed8ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.756+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee81db820 con 0x7f5ee80b0770 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.757+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ed8001ac0 con 0x7f5ee80b0770 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.757+0000 7f5ecffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.757+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5ed8003a40 con 0x7f5ee80b0770 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.757+0000 7f5ecffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.757+0000 7f5ecffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.757+0000 7f5ecffff640 10 monclient: dump: 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:19.766 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.757+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f5ed8003c60 con 0x7f5ee80b0770 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.757+0000 7f5ee57fa640 4 set_mon_vals no callback set 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.758+0000 7f5ee67fc640 10 monclient: get_auth_request con 0x7f5ed406dd70 auth_method 0 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.758+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f5ed8085ce0 con 0x7f5ee80b0770 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.758+0000 7f5ecffff640 10 client.4475.objecter ms_dispatch 0x7f5ee8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.758+0000 7f5ecffff640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:19.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.758+0000 7f5ecffff640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:19.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.758+0000 7f5ecffff640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:42:19.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.758+0000 7f5ecffff640 10 client.4475.objecter ms_handle_connect 0x7f5ed406dd70 2024-03-29T11:42:19.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.765+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ed8050610 con 0x7f5ee80b0770 2024-03-29T11:42:19.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.765+0000 7f5ecffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:19.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:19.765+0000 7f5ecffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:20.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.058+0000 7f5eed8ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:20.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.059+0000 7f5eed8ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:20.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.059+0000 7f5eed8ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:20.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.059+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ee8075f10 con 0x7f5ee80b0770 2024-03-29T11:42:20.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.059+0000 7f5ecffff640 1 -- 172.21.15.100:0/339557220 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f5ed8052540 con 0x7f5ee80b0770 2024-03-29T11:42:20.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.059+0000 7f5ecffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:20.062 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.059+0000 7f5ecffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:20.063 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:42:20.063 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:20.065 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.062+0000 7f5eed8ae640 10 librados: watch_flush enter 2024-03-29T11:42:20.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.063+0000 7f5eed8ae640 10 librados: watch_flush exit 2024-03-29T11:42:20.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.063+0000 7f5eed8ae640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:42:20.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.063+0000 7f5eed8ae640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:42:20.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.063+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 >> v1:172.21.15.100:6816/3220289657 conn(0x7f5ed406dd70 legacy=0x7f5ed4070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.063+0000 7f5eed8ae640 10 monclient: shutdown 2024-03-29T11:42:20.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.063+0000 7f5eed8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:20.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.063+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 >> v1:172.21.15.100:6790/0 conn(0x7f5ee80b0770 legacy=0x7f5ee81dbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.063+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 shutdown_connections 2024-03-29T11:42:20.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.063+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 >> 172.21.15.100:0/339557220 conn(0x7f5ee806b660 msgr2=0x7f5ee80adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:20.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.063+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 shutdown_connections 2024-03-29T11:42:20.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.064+0000 7f5eed8ae640 1 -- 172.21.15.100:0/339557220 wait complete. 2024-03-29T11:42:20.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:20.064+0000 7f5eed8ae640 1 librados: shutdown 2024-03-29T11:42:20.080 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:26.888 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.032+0000 7f44c8d10640 10 monclient: get_monmap_and_config 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.032+0000 7f44c8d10640 10 monclient: build_initial_monmap 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.032+0000 7f44c8d10640 10 monclient: monmap: 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:42:27.033877+0000 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:42:27.033877+0000 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:27.036 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:42:27.037 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:42:27.037 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:27.039 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.037+0000 7f44c8d10640 1 Processor -- start 2024-03-29T11:42:27.040 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.037+0000 7f44c8d10640 1 -- start start 2024-03-29T11:42:27.040 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.037+0000 7f44c8d10640 10 monclient: init 2024-03-29T11:42:27.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.038+0000 7f44c8d10640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:27.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.038+0000 7f44c8d10640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:27.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.038+0000 7f44c8d10640 10 monclient(hunting): picked mon.noname-a con 0x7f44bc0389e0 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:27.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.038+0000 7f44c8d10640 10 monclient(hunting): picked mon.noname-b con 0x7f44bc038dc0 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:27.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.038+0000 7f44c8d10640 10 monclient(hunting): picked mon.noname-c con 0x7f44bc030660 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:27.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.038+0000 7f44c8d10640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44bc0c54c0 con 0x7f44bc0389e0 2024-03-29T11:42:27.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.038+0000 7f44c8d10640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44bc036430 con 0x7f44bc030660 2024-03-29T11:42:27.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.038+0000 7f44c8d10640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44bc039840 con 0x7f44bc038dc0 2024-03-29T11:42:27.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.038+0000 7f44c8d10640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:27.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.038+0000 7f44c8d10640 10 monclient(hunting): authenticate will time out at 1267.233231s 2024-03-29T11:42:27.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.041+0000 7f44c3fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f44bc030660 0x7f44bc0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:48571/0 (socket says 172.21.15.100:48571) 2024-03-29T11:42:27.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.041+0000 7f44c3fff640 1 -- 172.21.15.100:0/1967463563 learned_addr learned my addr 172.21.15.100:0/1967463563 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:27.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775216650 0 0) 0x7f44bc039840 con 0x7f44bc038dc0 2024-03-29T11:42:27.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:27.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:27.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:27.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:42:27.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b4003520 con 0x7f44bc038dc0 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454061869 0 0) 0x7f44bc036430 con 0x7f44bc030660 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44bc039840 con 0x7f44bc030660 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.043+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 555363389 0 0) 0x7f44bc039840 con 0x7f44bc030660 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 >> v1:172.21.15.100:6790/0 conn(0x7f44bc038dc0 legacy=0x7f44bc0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 >> v1:172.21.15.100:6789/0 conn(0x7f44bc0389e0 legacy=0x7f44bc033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 1 monclient: found mon.noname-c 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:42:27.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc035690 con 0x7f44bc030660 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f44b8001610 con 0x7f44bc030660 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: dump: 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:27.049 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c8d10640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c8d10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44b80017e0 con 0x7f44bc030660 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 1 -- 172.21.15.100:0/1967463563 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f44b8001bf0 con 0x7f44bc030660 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c27fc640 10 monclient: dump: 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:27.050 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:27.051 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:27.051 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:27.051 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:27.051 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:27.051 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:27.051 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:27.051 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:27.051 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:27.051 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c8d10640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:27.051 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.044+0000 7f44c8d10640 4 set_mon_vals no callback set 2024-03-29T11:42:27.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.045+0000 7f44c8d10640 10 monclient: shutdown 2024-03-29T11:42:27.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.045+0000 7f44c8d10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:27.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.045+0000 7f44c8d10640 1 -- 172.21.15.100:0/1967463563 >> v1:172.21.15.191:6789/0 conn(0x7f44bc030660 legacy=0x7f44bc0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.045+0000 7f44c8d10640 1 -- 172.21.15.100:0/1967463563 shutdown_connections 2024-03-29T11:42:27.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.045+0000 7f44c8d10640 1 -- 172.21.15.100:0/1967463563 >> 172.21.15.100:0/1967463563 conn(0x7f44bc005c00 msgr2=0x7f44bc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:27.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.045+0000 7f44c8d10640 1 -- 172.21.15.100:0/1967463563 shutdown_connections 2024-03-29T11:42:27.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.045+0000 7f44c8d10640 1 -- 172.21.15.100:0/1967463563 wait complete. 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.046+0000 7f44c8d10640 10 monclient: build_initial_monmap 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.047+0000 7f44c8d10640 10 monclient: monmap: 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:27.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.047+0000 7f44c8d10640 1 librados: starting msgr at 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.047+0000 7f44c8d10640 1 librados: starting objecter 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.047+0000 7f44c8d10640 1 Processor -- start 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.048+0000 7f44c8d10640 1 -- start start 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.048+0000 7f44c8d10640 1 librados: setting wanted keys 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.048+0000 7f44c8d10640 1 librados: calling monclient init 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.048+0000 7f44c8d10640 10 monclient: init 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.049+0000 7f44c8d10640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.049+0000 7f44c8d10640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.049+0000 7f44c8d10640 10 monclient(hunting): picked mon.noname-c con 0x7f44bc030660 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.049+0000 7f44c8d10640 10 monclient(hunting): picked mon.noname-a con 0x7f44bc0389e0 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.050+0000 7f44c8d10640 10 monclient(hunting): picked mon.noname-b con 0x7f44bc0c6d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.050+0000 7f44c8d10640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44bc0c7720 con 0x7f44bc0389e0 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.050+0000 7f44c8d10640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44bc0c2f60 con 0x7f44bc0c6d70 2024-03-29T11:42:27.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.050+0000 7f44c8d10640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44bc0c7200 con 0x7f44bc030660 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.050+0000 7f44c8d10640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.050+0000 7f44c8d10640 10 monclient(hunting): authenticate will time out at 1267.245401s 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.050+0000 7f44c37fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f44bc030660 0x7f44bc0c7e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:56918/0 (socket says 172.21.15.100:56918) 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.050+0000 7f44c37fe640 1 -- 172.21.15.100:0/4067272026 learned_addr learned my addr 172.21.15.100:0/4067272026 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 client.?.objecter ms_handle_connect 0x7f44bc030660 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258940669 0 0) 0x7f44bc0c7200 con 0x7f44bc030660 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a8004eb0 con 0x7f44bc030660 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 client.?.objecter ms_handle_connect 0x7f44bc0389e0 2024-03-29T11:42:27.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:27.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602272818 0 0) 0x7f44bc0c7720 con 0x7f44bc0389e0 2024-03-29T11:42:27.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:27.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:27.065 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:27.065 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:42:27.065 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.052+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44bc0c7200 con 0x7f44bc0389e0 2024-03-29T11:42:27.065 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 client.?.objecter ms_handle_connect 0x7f44bc0c6d70 2024-03-29T11:42:27.065 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:27.065 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4126291384 0 0) 0x7f44a8004eb0 con 0x7f44bc030660 2024-03-29T11:42:27.065 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44bc0c7720 con 0x7f44bc030660 2024-03-29T11:42:27.065 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3954082537 0 0) 0x7f44bc0c7200 con 0x7f44bc0389e0 2024-03-29T11:42:27.065 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44a8004eb0 con 0x7f44bc0389e0 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f44b0001a30 con 0x7f44bc030660 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient(hunting): dump: 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:27.066 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f44c404fdd0 con 0x7f44bc0389e0 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient(hunting): dump: 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:27.067 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1923654635 0 0) 0x7f44bc0c7720 con 0x7f44bc030660 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 >> v1:172.21.15.191:6789/0 conn(0x7f44bc0c6d70 legacy=0x7f44bc0c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 >> v1:172.21.15.100:6789/0 conn(0x7f44bc0389e0 legacy=0x7f44bc15fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 monclient: found mon.c 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc02f340 con 0x7f44bc030660 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient: _finish_auth 0 2024-03-29T11:42:27.068 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:27.069 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:27.069 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44c8d10640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:42:27.069 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44c8d10640 10 monclient: _renew_subs 2024-03-29T11:42:27.069 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44c8d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:27.069 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.053+0000 7f44c8d10640 1 -- 172.21.15.100:0/4067272026 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44bc02f590 con 0x7f44bc030660 2024-03-29T11:42:27.069 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.054+0000 7f44c8d10640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:27.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.054+0000 7f44c8d10640 10 monclient: _renew_subs 2024-03-29T11:42:27.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.054+0000 7f44c8d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:27.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.054+0000 7f44c8d10640 1 -- 172.21.15.100:0/4067272026 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44bc160620 con 0x7f44bc030660 2024-03-29T11:42:27.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.054+0000 7f44c8d10640 1 librados: init done 2024-03-29T11:42:27.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.055+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44b0001cf0 con 0x7f44bc030660 2024-03-29T11:42:27.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.055+0000 7f44affff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:27.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.055+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f44b0003a60 con 0x7f44bc030660 2024-03-29T11:42:27.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.055+0000 7f44affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:27.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.055+0000 7f44affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:27.070 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.055+0000 7f44c17fa640 4 set_mon_vals no callback set 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.055+0000 7f44affff640 10 monclient: dump: 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.057+0000 7f44adffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:27.071 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.057+0000 7f44adffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:27.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.057+0000 7f44adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:27.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.057+0000 7f44adffb640 1 -- 172.21.15.100:0/4067272026 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a4005080 con 0x7f44bc030660 2024-03-29T11:42:27.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.057+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f44b0003c80 con 0x7f44bc030660 2024-03-29T11:42:27.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.058+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f44b0084b50 con 0x7f44bc030660 2024-03-29T11:42:27.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.058+0000 7f44c2ffd640 10 monclient: get_auth_request con 0x7f44a806ddf0 auth_method 0 2024-03-29T11:42:27.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.058+0000 7f44affff640 10 client.4481.objecter ms_dispatch 0x7f44c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:27.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.058+0000 7f44affff640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:27.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.058+0000 7f44affff640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:27.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.058+0000 7f44affff640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:42:27.072 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.058+0000 7f44affff640 10 client.4481.objecter ms_handle_connect 0x7f44a806ddf0 2024-03-29T11:42:27.073 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.066+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44b0050430 con 0x7f44bc030660 2024-03-29T11:42:27.073 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.066+0000 7f44affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:27.073 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.066+0000 7f44affff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:27.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.389+0000 7f44adffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:27.393 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.389+0000 7f44adffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:27.393 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.389+0000 7f44adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:27.393 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.389+0000 7f44adffb640 1 -- 172.21.15.100:0/4067272026 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44a4005330 con 0x7f44bc030660 2024-03-29T11:42:27.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.392+0000 7f44affff640 1 -- 172.21.15.100:0/4067272026 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f44b0052360 con 0x7f44bc030660 2024-03-29T11:42:27.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.392+0000 7f44affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:27.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.392+0000 7f44affff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:27.396 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:42:27.396 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:27.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.397+0000 7f44adffb640 10 librados: watch_flush enter 2024-03-29T11:42:27.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.397+0000 7f44adffb640 10 librados: watch_flush exit 2024-03-29T11:42:27.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.397+0000 7f44adffb640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:42:27.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.397+0000 7f44adffb640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:42:27.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.397+0000 7f44adffb640 1 -- 172.21.15.100:0/4067272026 >> v1:172.21.15.100:6816/3220289657 conn(0x7f44a806ddf0 legacy=0x7f44a80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.397+0000 7f44adffb640 10 monclient: shutdown 2024-03-29T11:42:27.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.397+0000 7f44adffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:27.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.397+0000 7f44adffb640 1 -- 172.21.15.100:0/4067272026 >> v1:172.21.15.100:6790/0 conn(0x7f44bc030660 legacy=0x7f44bc0c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:27.401 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.397+0000 7f44adffb640 1 -- 172.21.15.100:0/4067272026 shutdown_connections 2024-03-29T11:42:27.402 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.397+0000 7f44adffb640 1 -- 172.21.15.100:0/4067272026 >> 172.21.15.100:0/4067272026 conn(0x7f44bc005c00 msgr2=0x7f44bc034b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:27.402 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.398+0000 7f44adffb640 1 -- 172.21.15.100:0/4067272026 shutdown_connections 2024-03-29T11:42:27.402 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.398+0000 7f44adffb640 1 -- 172.21.15.100:0/4067272026 wait complete. 2024-03-29T11:42:27.402 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:27.398+0000 7f44adffb640 1 librados: shutdown 2024-03-29T11:42:27.418 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:34.225 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:34.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.375+0000 7fbff633b640 10 monclient: get_monmap_and_config 2024-03-29T11:42:34.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.375+0000 7fbff633b640 10 monclient: build_initial_monmap 2024-03-29T11:42:34.381 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.375+0000 7fbff633b640 10 monclient: monmap: 2024-03-29T11:42:34.381 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:34.381 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:34.382 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:42:34.376554+0000 2024-03-29T11:42:34.382 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:42:34.376554+0000 2024-03-29T11:42:34.382 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:34.382 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:34.382 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:34.382 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:42:34.382 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:42:34.382 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.378+0000 7fbff633b640 1 Processor -- start 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.379+0000 7fbff633b640 1 -- start start 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.379+0000 7fbff633b640 10 monclient: init 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.380+0000 7fbff633b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.380+0000 7fbff633b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.380+0000 7fbff633b640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe8034850 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.380+0000 7fbff633b640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe802a640 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.380+0000 7fbff633b640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe80327d0 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.380+0000 7fbff633b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe802c820 con 0x7fbfe80327d0 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.380+0000 7fbff633b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe802a120 con 0x7fbfe8034850 2024-03-29T11:42:34.383 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.380+0000 7fbff633b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe80c75b0 con 0x7fbfe802a640 2024-03-29T11:42:34.384 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.380+0000 7fbff633b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:34.384 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.380+0000 7fbff633b640 10 monclient(hunting): authenticate will time out at 1274.575391s 2024-03-29T11:42:34.386 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.383+0000 7fbff4b38640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fbfe802a640 0x7fbfe80320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:44862/0 (socket says 172.21.15.100:44862) 2024-03-29T11:42:34.386 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.383+0000 7fbff4b38640 1 -- 172.21.15.100:0/1033882939 learned_addr learned my addr 172.21.15.100:0/1033882939 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:34.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.385+0000 7fbfe7fff640 1 -- 172.21.15.100:0/1033882939 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172493881 0 0) 0x7fbfe80c75b0 con 0x7fbfe802a640 2024-03-29T11:42:34.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.385+0000 7fbfe7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:34.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.385+0000 7fbfe7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:34.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.385+0000 7fbfe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:34.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.385+0000 7fbfe7fff640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:42:34.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.385+0000 7fbfe7fff640 1 -- 172.21.15.100:0/1033882939 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfdc003520 con 0x7fbfe802a640 2024-03-29T11:42:34.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 1 -- 172.21.15.100:0/1033882939 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938665370 0 0) 0x7fbfdc003520 con 0x7fbfe802a640 2024-03-29T11:42:34.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 1 -- 172.21.15.100:0/1033882939 >> v1:172.21.15.191:6789/0 conn(0x7fbfe8034850 legacy=0x7fbfe80c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:34.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 1 -- 172.21.15.100:0/1033882939 >> v1:172.21.15.100:6789/0 conn(0x7fbfe80327d0 legacy=0x7fbfe80cf240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 1 monclient: found mon.noname-b 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 1 -- 172.21.15.100:0/1033882939 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe8033110 con 0x7fbfe802a640 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 1 -- 172.21.15.100:0/1033882939 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fbfe0001690 con 0x7fbfe802a640 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: dump: 2024-03-29T11:42:34.390 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbff633b640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbff633b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:34.391 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 1 -- 172.21.15.100:0/1033882939 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfe00018f0 con 0x7fbfe802a640 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 1 -- 172.21.15.100:0/1033882939 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fbfe0001d40 con 0x7fbfe802a640 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbfe7fff640 10 monclient: dump: 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:34.392 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:34.393 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:34.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbff633b640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:34.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.386+0000 7fbff633b640 4 set_mon_vals no callback set 2024-03-29T11:42:34.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.387+0000 7fbff633b640 10 monclient: shutdown 2024-03-29T11:42:34.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.387+0000 7fbff633b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:34.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.387+0000 7fbff633b640 1 -- 172.21.15.100:0/1033882939 >> v1:172.21.15.100:6790/0 conn(0x7fbfe802a640 legacy=0x7fbfe80320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:34.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.387+0000 7fbff633b640 1 -- 172.21.15.100:0/1033882939 shutdown_connections 2024-03-29T11:42:34.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.387+0000 7fbff633b640 1 -- 172.21.15.100:0/1033882939 >> 172.21.15.100:0/1033882939 conn(0x7fbfe80058f0 msgr2=0x7fbfe8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:34.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.388+0000 7fbff633b640 1 -- 172.21.15.100:0/1033882939 shutdown_connections 2024-03-29T11:42:34.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.388+0000 7fbff633b640 1 -- 172.21.15.100:0/1033882939 wait complete. 2024-03-29T11:42:34.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.389+0000 7fbff633b640 10 monclient: build_initial_monmap 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.389+0000 7fbff633b640 10 monclient: monmap: 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.389+0000 7fbff633b640 1 librados: starting msgr at 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.389+0000 7fbff633b640 1 librados: starting objecter 2024-03-29T11:42:34.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.389+0000 7fbff633b640 1 Processor -- start 2024-03-29T11:42:34.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.389+0000 7fbff633b640 1 -- start start 2024-03-29T11:42:34.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.389+0000 7fbff633b640 1 librados: setting wanted keys 2024-03-29T11:42:34.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.389+0000 7fbff633b640 1 librados: calling monclient init 2024-03-29T11:42:34.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.389+0000 7fbff633b640 10 monclient: init 2024-03-29T11:42:34.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.390+0000 7fbff633b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:34.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.390+0000 7fbff633b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:34.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.390+0000 7fbff633b640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe802a640 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:34.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.390+0000 7fbff633b640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe80327d0 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:34.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.390+0000 7fbff633b640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe8034850 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:34.398 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.390+0000 7fbff633b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe8158690 con 0x7fbfe8034850 2024-03-29T11:42:34.398 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.390+0000 7fbff633b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe81582d0 con 0x7fbfe802a640 2024-03-29T11:42:34.398 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.390+0000 7fbff633b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe80bc860 con 0x7fbfe80327d0 2024-03-29T11:42:34.398 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.390+0000 7fbff633b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:34.398 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.390+0000 7fbff633b640 10 monclient(hunting): authenticate will time out at 1274.585721s 2024-03-29T11:42:34.398 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.391+0000 7fbff4b38640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fbfe80327d0 0x7fbfe80bba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:45908/0 (socket says 172.21.15.100:45908) 2024-03-29T11:42:34.399 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.391+0000 7fbff4b38640 1 -- 172.21.15.100:0/3198104826 learned_addr learned my addr 172.21.15.100:0/3198104826 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:34.399 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.391+0000 7fbfe5ffb640 10 client.?.objecter ms_handle_connect 0x7fbfe80327d0 2024-03-29T11:42:34.399 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.391+0000 7fbfe5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:34.404 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.393+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568223155 0 0) 0x7fbfe80bc860 con 0x7fbfe80327d0 2024-03-29T11:42:34.404 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd4004eb0 con 0x7fbfe80327d0 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 92807689 0 0) 0x7fbfd4004eb0 con 0x7fbfe80327d0 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfe80bc860 con 0x7fbfe80327d0 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 10 client.?.objecter ms_handle_connect 0x7fbfe802a640 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fbfe00014b0 con 0x7fbfe80327d0 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:34.405 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.394+0000 7fbfe5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.395+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786198501 0 0) 0x7fbfe81582d0 con 0x7fbfe802a640 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.395+0000 7fbfe5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.395+0000 7fbfe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.395+0000 7fbfe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:34.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.395+0000 7fbfe5ffb640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.395+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd4004eb0 con 0x7fbfe802a640 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 client.?.objecter ms_handle_connect 0x7fbfe8034850 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501305735 0 0) 0x7fbfe8158690 con 0x7fbfe8034850 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe81582d0 con 0x7fbfe8034850 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 7042044 0 0) 0x7fbfd4004eb0 con 0x7fbfe802a640 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfe8158690 con 0x7fbfe802a640 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fbff004fdd0 con 0x7fbfe802a640 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:34.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:34.408 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4166750949 0 0) 0x7fbfe81582d0 con 0x7fbfe8034850 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfd4004eb0 con 0x7fbfe8034850 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fbfec0016f0 con 0x7fbfe8034850 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:34.409 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 720641452 0 0) 0x7fbfd4004eb0 con 0x7fbfe8034850 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 >> v1:172.21.15.100:6790/0 conn(0x7fbfe80327d0 legacy=0x7fbfe80bba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 >> v1:172.21.15.191:6789/0 conn(0x7fbfe802a640 legacy=0x7fbfe8158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 monclient: found mon.a 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe802cc40 con 0x7fbfe8034850 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:34.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.397+0000 7fbfe5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:34.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfec001c00 con 0x7fbfe8034850 2024-03-29T11:42:34.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbfe5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:34.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbfe6ffd640 4 set_mon_vals no callback set 2024-03-29T11:42:34.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fbfec003790 con 0x7fbfe8034850 2024-03-29T11:42:34.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbfe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:34.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbfe5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbfe5ffb640 10 monclient: dump: 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbff633b640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbff633b640 10 monclient: _renew_subs 2024-03-29T11:42:34.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbff633b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.398+0000 7fbff633b640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe80314c0 con 0x7fbfe8034850 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.399+0000 7fbff633b640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.399+0000 7fbff633b640 10 monclient: _renew_subs 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.399+0000 7fbff633b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.399+0000 7fbff633b640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe80c05a0 con 0x7fbfe8034850 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.399+0000 7fbff633b640 1 librados: init done 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.400+0000 7fbff633b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.400+0000 7fbff633b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.400+0000 7fbff633b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.400+0000 7fbff633b640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe8006270 con 0x7fbfe8034850 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.401+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fbfec019760 con 0x7fbfe8034850 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.401+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fbfec083610 con 0x7fbfe8034850 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.401+0000 7fbfe5ffb640 10 client.4419.objecter ms_dispatch 0x7fbff0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.401+0000 7fbfe5ffb640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:34.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.401+0000 7fbfe5ffb640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:34.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.402+0000 7fbfe5ffb640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-29T11:42:34.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.402+0000 7fbff5339640 10 monclient: get_auth_request con 0x7fbfd406e230 auth_method 0 2024-03-29T11:42:34.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.402+0000 7fbfe5ffb640 10 client.4419.objecter ms_handle_connect 0x7fbfd406e230 2024-03-29T11:42:34.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.412+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbfe8006270 con 0x7fbfe8034850 2024-03-29T11:42:34.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.412+0000 7fbfe5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:34.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.412+0000 7fbfe5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:34.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.741+0000 7fbff633b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:34.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.741+0000 7fbff633b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:34.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.741+0000 7fbff633b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:42:34.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.741+0000 7fbff633b640 1 -- 172.21.15.100:0/3198104826 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfe8006270 con 0x7fbfe8034850 2024-03-29T11:42:34.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.743+0000 7fbfe5ffb640 1 -- 172.21.15.100:0/3198104826 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fbfe8006270 con 0x7fbfe8034850 2024-03-29T11:42:34.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.743+0000 7fbfe5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:34.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.743+0000 7fbfe5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:34.746 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:42:34.747 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.747+0000 7fbff633b640 10 librados: watch_flush enter 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.747+0000 7fbff633b640 10 librados: watch_flush exit 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.747+0000 7fbff633b640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.747+0000 7fbff633b640 10 client.4419.objecter successfully canceled tick 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.747+0000 7fbff633b640 1 -- 172.21.15.100:0/3198104826 >> v1:172.21.15.100:6816/3220289657 conn(0x7fbfd406e230 legacy=0x7fbfd40706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.747+0000 7fbff633b640 10 monclient: shutdown 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.747+0000 7fbff633b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.747+0000 7fbff633b640 1 -- 172.21.15.100:0/3198104826 >> v1:172.21.15.100:6789/0 conn(0x7fbfe8034850 legacy=0x7fbfe80bc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.747+0000 7fbff633b640 1 -- 172.21.15.100:0/3198104826 shutdown_connections 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.747+0000 7fbff633b640 1 -- 172.21.15.100:0/3198104826 >> 172.21.15.100:0/3198104826 conn(0x7fbfe80058f0 msgr2=0x7fbfe80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.748+0000 7fbff633b640 1 -- 172.21.15.100:0/3198104826 shutdown_connections 2024-03-29T11:42:34.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.748+0000 7fbff633b640 1 -- 172.21.15.100:0/3198104826 wait complete. 2024-03-29T11:42:34.754 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:34.751+0000 7fbff633b640 1 librados: shutdown 2024-03-29T11:42:34.772 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:41.578 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:41.735 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.731+0000 7fd5daa7b640 10 monclient: get_monmap_and_config 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.731+0000 7fd5daa7b640 10 monclient: build_initial_monmap 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.731+0000 7fd5daa7b640 10 monclient: monmap: 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:42:41.732910+0000 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:42:41.732910+0000 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:42:41.736 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:41.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.733+0000 7fd5daa7b640 1 Processor -- start 2024-03-29T11:42:41.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.733+0000 7fd5daa7b640 1 -- start start 2024-03-29T11:42:41.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.733+0000 7fd5daa7b640 10 monclient: init 2024-03-29T11:42:41.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.734+0000 7fd5daa7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:41.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.734+0000 7fd5daa7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:41.737 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.734+0000 7fd5daa7b640 10 monclient(hunting): picked mon.noname-a con 0x7fd5d40a9ac0 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:41.738 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.734+0000 7fd5daa7b640 10 monclient(hunting): picked mon.noname-b con 0x7fd5d413ee90 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:41.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.738+0000 7fd5daa7b640 10 monclient(hunting): picked mon.noname-c con 0x7fd5d40af900 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:41.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.738+0000 7fd5d9018640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd5d40af900 0x7fd5d4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:39057/0 (socket says 172.21.15.100:39057) 2024-03-29T11:42:41.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.738+0000 7fd5d9018640 1 -- 172.21.15.100:0/3131362333 learned_addr learned my addr 172.21.15.100:0/3131362333 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:41.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.738+0000 7fd5daa7b640 1 -- 172.21.15.100:0/3131362333 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d40afd10 con 0x7fd5d40a9ac0 2024-03-29T11:42:41.741 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.738+0000 7fd5daa7b640 1 -- 172.21.15.100:0/3131362333 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d40af6f0 con 0x7fd5d40af900 2024-03-29T11:42:41.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.738+0000 7fd5daa7b640 1 -- 172.21.15.100:0/3131362333 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d40aac80 con 0x7fd5d413ee90 2024-03-29T11:42:41.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.739+0000 7fd5daa7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:41.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.739+0000 7fd5d37fe640 1 -- 172.21.15.100:0/3131362333 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180035467 0 0) 0x7fd5d40af6f0 con 0x7fd5d40af900 2024-03-29T11:42:41.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.739+0000 7fd5daa7b640 10 monclient(hunting): authenticate will time out at 1281.934388s 2024-03-29T11:42:41.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.739+0000 7fd5d37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:41.744 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.739+0000 7fd5d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.739+0000 7fd5d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.739+0000 7fd5d37fe640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.739+0000 7fd5d37fe640 1 -- 172.21.15.100:0/3131362333 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c4003520 con 0x7fd5d40af900 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.740+0000 7fd5d37fe640 1 -- 172.21.15.100:0/3131362333 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4030729579 0 0) 0x7fd5c4003520 con 0x7fd5d40af900 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.740+0000 7fd5d37fe640 1 -- 172.21.15.100:0/3131362333 >> v1:172.21.15.100:6790/0 conn(0x7fd5d413ee90 legacy=0x7fd5d413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.740+0000 7fd5d37fe640 1 -- 172.21.15.100:0/3131362333 >> v1:172.21.15.100:6789/0 conn(0x7fd5d40a9ac0 legacy=0x7fd5d40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.740+0000 7fd5d37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.740+0000 7fd5d37fe640 1 monclient: found mon.noname-c 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.740+0000 7fd5d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.740+0000 7fd5d37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.740+0000 7fd5d37fe640 1 -- 172.21.15.100:0/3131362333 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d40aaf60 con 0x7fd5d40af900 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.741+0000 7fd5d37fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.741+0000 7fd5d37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:41.745 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.741+0000 7fd5d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:41.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.741+0000 7fd5d37fe640 1 -- 172.21.15.100:0/3131362333 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd5cc001a10 con 0x7fd5d40af900 2024-03-29T11:42:41.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.741+0000 7fd5daa7b640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:42:41.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.741+0000 7fd5daa7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:41.746 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.741+0000 7fd5d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.741+0000 7fd5d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.742+0000 7fd5d37fe640 10 monclient: dump: 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.742+0000 7fd5d37fe640 1 -- 172.21.15.100:0/3131362333 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5cc001ca0 con 0x7fd5d40af900 2024-03-29T11:42:41.748 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.742+0000 7fd5daa7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.742+0000 7fd5d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.742+0000 7fd5d37fe640 1 -- 172.21.15.100:0/3131362333 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd5cc002070 con 0x7fd5d40af900 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.742+0000 7fd5daa7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.742+0000 7fd5daa7b640 4 set_mon_vals no callback set 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.743+0000 7fd5d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.743+0000 7fd5d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.743+0000 7fd5d37fe640 10 monclient: dump: 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:41.749 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:41.750 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:41.750 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:41.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.743+0000 7fd5daa7b640 10 monclient: shutdown 2024-03-29T11:42:41.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.743+0000 7fd5daa7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:41.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.744+0000 7fd5daa7b640 1 -- 172.21.15.100:0/3131362333 >> v1:172.21.15.191:6789/0 conn(0x7fd5d40af900 legacy=0x7fd5d4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:41.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.744+0000 7fd5daa7b640 1 -- 172.21.15.100:0/3131362333 shutdown_connections 2024-03-29T11:42:41.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.744+0000 7fd5daa7b640 1 -- 172.21.15.100:0/3131362333 >> 172.21.15.100:0/3131362333 conn(0x7fd5d40b35e0 msgr2=0x7fd5d40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:41.750 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.745+0000 7fd5daa7b640 1 -- 172.21.15.100:0/3131362333 shutdown_connections 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.747+0000 7fd5daa7b640 1 -- 172.21.15.100:0/3131362333 wait complete. 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.748+0000 7fd5daa7b640 10 monclient: build_initial_monmap 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.748+0000 7fd5daa7b640 10 monclient: monmap: 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:42:41.751 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:42:41.752 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:41.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.748+0000 7fd5daa7b640 1 librados: starting msgr at 2024-03-29T11:42:41.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.748+0000 7fd5daa7b640 1 librados: starting objecter 2024-03-29T11:42:41.752 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.748+0000 7fd5daa7b640 1 Processor -- start 2024-03-29T11:42:41.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.748+0000 7fd5daa7b640 1 -- start start 2024-03-29T11:42:41.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.749+0000 7fd5daa7b640 1 librados: setting wanted keys 2024-03-29T11:42:41.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.749+0000 7fd5daa7b640 1 librados: calling monclient init 2024-03-29T11:42:41.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.749+0000 7fd5daa7b640 10 monclient: init 2024-03-29T11:42:41.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.750+0000 7fd5daa7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:41.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.750+0000 7fd5daa7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:41.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.750+0000 7fd5daa7b640 10 monclient(hunting): picked mon.noname-b con 0x7fd5d40a9ac0 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:41.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.750+0000 7fd5daa7b640 10 monclient(hunting): picked mon.noname-a con 0x7fd5d40af900 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:41.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.753+0000 7fd5daa7b640 10 monclient(hunting): picked mon.noname-c con 0x7fd5d413ee90 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:41.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.753+0000 7fd5daa7b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d41395e0 con 0x7fd5d40af900 2024-03-29T11:42:41.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.753+0000 7fd5daa7b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d41379b0 con 0x7fd5d40a9ac0 2024-03-29T11:42:41.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.753+0000 7fd5daa7b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d4137b40 con 0x7fd5d413ee90 2024-03-29T11:42:41.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.753+0000 7fd5daa7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:41.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.753+0000 7fd5daa7b640 10 monclient(hunting): authenticate will time out at 1281.948197s 2024-03-29T11:42:41.757 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.754+0000 7fd5d9018640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fd5d413ee90 0x7fd5d41e0af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:41860/0 (socket says 172.21.15.100:41860) 2024-03-29T11:42:41.757 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.754+0000 7fd5d9018640 1 -- 172.21.15.100:0/433310877 learned_addr learned my addr 172.21.15.100:0/433310877 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:41.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.754+0000 7fd5d17fa640 10 client.?.objecter ms_handle_connect 0x7fd5d413ee90 2024-03-29T11:42:41.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.755+0000 7fd5d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:41.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.755+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318069255 0 0) 0x7fd5d4137b40 con 0x7fd5d413ee90 2024-03-29T11:42:41.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.755+0000 7fd5d17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:41.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.755+0000 7fd5d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:41.758 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.755+0000 7fd5d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:41.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.755+0000 7fd5d17fa640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:42:41.759 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.755+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b8004eb0 con 0x7fd5d413ee90 2024-03-29T11:42:41.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.756+0000 7fd5d17fa640 10 client.?.objecter ms_handle_connect 0x7fd5d40a9ac0 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.756+0000 7fd5d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.756+0000 7fd5d17fa640 10 client.?.objecter ms_handle_connect 0x7fd5d40af900 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.756+0000 7fd5d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.756+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199549502 0 0) 0x7fd5d41379b0 con 0x7fd5d40a9ac0 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.756+0000 7fd5d17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.756+0000 7fd5d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.756+0000 7fd5d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d4137b40 con 0x7fd5d40a9ac0 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665921265 0 0) 0x7fd5d41395e0 con 0x7fd5d40af900 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:41.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d41379b0 con 0x7fd5d40af900 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336286878 0 0) 0x7fd5b8004eb0 con 0x7fd5d413ee90 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5d41395e0 con 0x7fd5d413ee90 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.757+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd5cc001910 con 0x7fd5d413ee90 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 10 monclient(hunting): dump: 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:41.765 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2522775776 0 0) 0x7fd5d4137b40 con 0x7fd5d40a9ac0 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5b8004eb0 con 0x7fd5d40a9ac0 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3348872528 0 0) 0x7fd5d41395e0 con 0x7fd5d413ee90 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 >> v1:172.21.15.191:6789/0 conn(0x7fd5d40a9ac0 legacy=0x7fd5d41d87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 >> v1:172.21.15.100:6789/0 conn(0x7fd5d40af900 legacy=0x7fd5d41357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 1 monclient: found mon.c 2024-03-29T11:42:41.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.758+0000 7fd5d17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d413a6e0 con 0x7fd5d413ee90 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5d17fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5d17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5cc001a70 con 0x7fd5d413ee90 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd5cc005110 con 0x7fd5d413ee90 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5d17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5d17fa640 10 monclient: dump: 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:41.767 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5daa7b640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5daa7b640 10 monclient: _renew_subs 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.759+0000 7fd5daa7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.760+0000 7fd5daa7b640 1 -- 172.21.15.100:0/433310877 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d413a930 con 0x7fd5d413ee90 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.760+0000 7fd5daa7b640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.760+0000 7fd5daa7b640 10 monclient: _renew_subs 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.760+0000 7fd5daa7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.760+0000 7fd5daa7b640 1 -- 172.21.15.100:0/433310877 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d41360a0 con 0x7fd5d413ee90 2024-03-29T11:42:41.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.760+0000 7fd5daa7b640 1 librados: init done 2024-03-29T11:42:41.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.761+0000 7fd5d2ffd640 4 set_mon_vals no callback set 2024-03-29T11:42:41.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.762+0000 7fd5daa7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:41.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.762+0000 7fd5daa7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:41.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.762+0000 7fd5daa7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:41.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.762+0000 7fd5daa7b640 1 -- 172.21.15.100:0/433310877 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b4005080 con 0x7fd5d413ee90 2024-03-29T11:42:41.770 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.768+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fd5cc01b0e0 con 0x7fd5d413ee90 2024-03-29T11:42:41.772 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.769+0000 7fd5d8817640 10 monclient: get_auth_request con 0x7fd5b806df80 auth_method 0 2024-03-29T11:42:41.773 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.770+0000 7fd5d17fa640 10 client.4493.objecter ms_handle_connect 0x7fd5b806df80 2024-03-29T11:42:41.773 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.770+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fd5cc084db0 con 0x7fd5d413ee90 2024-03-29T11:42:41.773 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.770+0000 7fd5d17fa640 10 client.4493.objecter ms_dispatch 0x7fd5d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:41.773 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.770+0000 7fd5d17fa640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:41.773 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.770+0000 7fd5d17fa640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:41.773 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.770+0000 7fd5d17fa640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:42:41.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.774+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5cc050690 con 0x7fd5d413ee90 2024-03-29T11:42:41.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.774+0000 7fd5d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:41.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:41.774+0000 7fd5d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:42.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.148+0000 7fd5daa7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:42.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.148+0000 7fd5daa7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:42.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.148+0000 7fd5daa7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:42.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.148+0000 7fd5daa7b640 1 -- 172.21.15.100:0/433310877 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5b4005330 con 0x7fd5d413ee90 2024-03-29T11:42:42.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.150+0000 7fd5d17fa640 1 -- 172.21.15.100:0/433310877 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fd5cc0525c0 con 0x7fd5d413ee90 2024-03-29T11:42:42.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.150+0000 7fd5d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:42.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.150+0000 7fd5d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:42.155 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:42:42.155 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:42.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.153+0000 7fd5daa7b640 10 librados: watch_flush enter 2024-03-29T11:42:42.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.153+0000 7fd5daa7b640 10 librados: watch_flush exit 2024-03-29T11:42:42.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.153+0000 7fd5daa7b640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:42:42.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.153+0000 7fd5daa7b640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:42:42.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.153+0000 7fd5daa7b640 1 -- 172.21.15.100:0/433310877 >> v1:172.21.15.100:6816/3220289657 conn(0x7fd5b806df80 legacy=0x7fd5b8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:42.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.153+0000 7fd5daa7b640 10 monclient: shutdown 2024-03-29T11:42:42.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.153+0000 7fd5daa7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:42.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.153+0000 7fd5daa7b640 1 -- 172.21.15.100:0/433310877 >> v1:172.21.15.100:6790/0 conn(0x7fd5d413ee90 legacy=0x7fd5d41e0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:42.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.154+0000 7fd5daa7b640 1 -- 172.21.15.100:0/433310877 shutdown_connections 2024-03-29T11:42:42.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.154+0000 7fd5daa7b640 1 -- 172.21.15.100:0/433310877 >> 172.21.15.100:0/433310877 conn(0x7fd5d40b35e0 msgr2=0x7fd5d40b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:42.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.156+0000 7fd5daa7b640 1 -- 172.21.15.100:0/433310877 shutdown_connections 2024-03-29T11:42:42.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.156+0000 7fd5daa7b640 1 -- 172.21.15.100:0/433310877 wait complete. 2024-03-29T11:42:42.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:42.156+0000 7fd5daa7b640 1 librados: shutdown 2024-03-29T11:42:42.180 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:48.988 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:49.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.120+0000 7fa6c21a8640 10 monclient: get_monmap_and_config 2024-03-29T11:42:49.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.120+0000 7fa6c21a8640 10 monclient: build_initial_monmap 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.120+0000 7fa6c21a8640 10 monclient: monmap: 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:42:49.121347+0000 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:42:49.121347+0000 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:42:49.124 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:49.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.124+0000 7fa6c21a8640 1 Processor -- start 2024-03-29T11:42:49.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.124+0000 7fa6c21a8640 1 -- start start 2024-03-29T11:42:49.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.124+0000 7fa6c21a8640 10 monclient: init 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.125+0000 7fa6c21a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.125+0000 7fa6c21a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.125+0000 7fa6c21a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6bc1422b0 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.125+0000 7fa6c21a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa6bc142690 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.125+0000 7fa6baffd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fa6bc142690 0x7fa6bc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:36050/0 (socket says 172.21.15.100:36050) 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.125+0000 7fa6bbfff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fa6bc0b17d0 0x7fa6bc0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:60976/0 (socket says 172.21.15.100:60976) 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.125+0000 7fa6bbfff640 1 -- 172.21.15.100:0/751792192 learned_addr learned my addr 172.21.15.100:0/751792192 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.125+0000 7fa6c21a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa6bc0b17d0 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6c21a8640 1 -- 172.21.15.100:0/751792192 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc142c80 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6c21a8640 1 -- 172.21.15.100:0/751792192 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc0aae40 con 0x7fa6bc1422b0 2024-03-29T11:42:49.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6c21a8640 1 -- 172.21.15.100:0/751792192 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc13fbd0 con 0x7fa6bc142690 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6c21a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6c21a8640 10 monclient(hunting): authenticate will time out at 1289.321542s 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845083350 0 0) 0x7fa6bc142c80 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6ba7fc640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a4003520 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591460872 0 0) 0x7fa6bc13fbd0 con 0x7fa6bc142690 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.126+0000 7fa6ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.127+0000 7fa6ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:49.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.127+0000 7fa6ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:49.133 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.127+0000 7fa6ba7fc640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:42:49.133 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.127+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6bc142c80 con 0x7fa6bc142690 2024-03-29T11:42:49.133 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.127+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 41811205 0 0) 0x7fa6a4003520 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.133 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.127+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 >> v1:172.21.15.100:6790/0 conn(0x7fa6bc142690 legacy=0x7fa6bc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:49.133 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.127+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 >> v1:172.21.15.191:6789/0 conn(0x7fa6bc1422b0 legacy=0x7fa6bc13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:49.144 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:49.144 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 1 monclient: found mon.noname-a 2024-03-29T11:42:49.144 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:49.144 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc14f920 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa6b4001a10 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6c21a8640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6c21a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.128+0000 7fa6ba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.129+0000 7fa6ba7fc640 10 monclient: dump: 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:49.145 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:49.146 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:49.146 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:49.146 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:49.146 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:49.146 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:49.146 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.129+0000 7fa6c21a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:49.146 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.129+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6b4001b70 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.146 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.129+0000 7fa6ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:49.146 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.129+0000 7fa6ba7fc640 1 -- 172.21.15.100:0/751792192 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa6b4001f00 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.146 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.129+0000 7fa6ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.129+0000 7fa6ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.129+0000 7fa6ba7fc640 10 monclient: dump: 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.129+0000 7fa6c21a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:49.147 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.129+0000 7fa6c21a8640 4 set_mon_vals no callback set 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.130+0000 7fa6c21a8640 10 monclient: shutdown 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.130+0000 7fa6c21a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.130+0000 7fa6c21a8640 1 -- 172.21.15.100:0/751792192 >> v1:172.21.15.100:6789/0 conn(0x7fa6bc0b17d0 legacy=0x7fa6bc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.130+0000 7fa6c21a8640 1 -- 172.21.15.100:0/751792192 shutdown_connections 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.130+0000 7fa6c21a8640 1 -- 172.21.15.100:0/751792192 >> 172.21.15.100:0/751792192 conn(0x7fa6bc066a00 msgr2=0x7fa6bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.130+0000 7fa6c21a8640 1 -- 172.21.15.100:0/751792192 shutdown_connections 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.131+0000 7fa6c21a8640 1 -- 172.21.15.100:0/751792192 wait complete. 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.131+0000 7fa6c21a8640 10 monclient: build_initial_monmap 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.132+0000 7fa6c21a8640 10 monclient: monmap: 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:42:49.148 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.132+0000 7fa6c21a8640 1 librados: starting msgr at 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.132+0000 7fa6c21a8640 1 librados: starting objecter 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.132+0000 7fa6c21a8640 1 Processor -- start 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.133+0000 7fa6c21a8640 1 -- start start 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.133+0000 7fa6c21a8640 1 librados: setting wanted keys 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.133+0000 7fa6c21a8640 1 librados: calling monclient init 2024-03-29T11:42:49.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.133+0000 7fa6c21a8640 10 monclient: init 2024-03-29T11:42:49.150 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.134+0000 7fa6c21a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:49.150 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.134+0000 7fa6c21a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:49.150 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.134+0000 7fa6c21a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6bc0b17d0 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:49.150 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.134+0000 7fa6c21a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa6bc1422b0 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:49.150 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6bb7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fa6bc0b17d0 0x7fa6bc0ad950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:39160/0 (socket says 172.21.15.100:39160) 2024-03-29T11:42:49.150 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6c21a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa6bc142690 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:49.150 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6bb7fe640 1 -- 172.21.15.100:0/4040266659 learned_addr learned my addr 172.21.15.100:0/4040266659 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:49.150 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc0af040 con 0x7fa6bc142690 2024-03-29T11:42:49.150 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc134c30 con 0x7fa6bc1422b0 2024-03-29T11:42:49.150 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc0b0560 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6c21a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6c21a8640 10 monclient(hunting): authenticate will time out at 1289.330518s 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 client.?.objecter ms_handle_connect 0x7fa6bc0b17d0 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 client.?.objecter ms_handle_connect 0x7fa6bc142690 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570860019 0 0) 0x7fa6bc0af040 con 0x7fa6bc142690 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:42:49.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a8004eb0 con 0x7fa6bc142690 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734339843 0 0) 0x7fa6bc0b0560 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.135+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6bc0af040 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2023886124 0 0) 0x7fa6bc0af040 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6bc0b0560 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa6ac0017b0 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 10 monclient(hunting): dump: 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:49.152 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4146259675 0 0) 0x7fa6a8004eb0 con 0x7fa6bc142690 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6bc0af040 con 0x7fa6bc142690 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa6b4001e80 con 0x7fa6bc142690 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:49.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 10 monclient(hunting): dump: 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3580826943 0 0) 0x7fa6bc0b0560 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.136+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 >> v1:172.21.15.191:6789/0 conn(0x7fa6bc1422b0 legacy=0x7fa6bc0af740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:49.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 >> v1:172.21.15.100:6789/0 conn(0x7fa6bc142690 legacy=0x7fa6bc0afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 1 monclient: found mon.c 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc0ad160 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6c21a8640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6c21a8640 10 monclient: _renew_subs 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6c21a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc0b0910 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6c21a8640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6c21a8640 10 monclient: _renew_subs 2024-03-29T11:42:49.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6c21a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc1dab70 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6c21a8640 1 librados: init done 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6ac001cc0 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa6ac004aa0 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6a3fff640 10 monclient: dump: 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:49.156 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.137+0000 7fa6b9ffb640 4 set_mon_vals no callback set 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.139+0000 7fa6c21a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.139+0000 7fa6c21a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.139+0000 7fa6c21a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.139+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69c005080 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.140+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fa6ac01aa70 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.140+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fa6ac0849d0 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.140+0000 7fa6a3fff640 10 client.4499.objecter ms_dispatch 0x7fa6bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:49.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.140+0000 7fa6a3fff640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:49.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.140+0000 7fa6a3fff640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:49.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.140+0000 7fa6a3fff640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:42:49.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.141+0000 7fa6baffd640 10 monclient: get_auth_request con 0x7fa6a806dda0 auth_method 0 2024-03-29T11:42:49.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.141+0000 7fa6a3fff640 10 client.4499.objecter ms_handle_connect 0x7fa6a806dda0 2024-03-29T11:42:49.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.146+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6ac0500e0 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.146+0000 7fa6a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:49.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.146+0000 7fa6a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:49.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.458+0000 7fa6c21a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:49.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.458+0000 7fa6c21a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:49.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.458+0000 7fa6c21a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:49.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.458+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa69c005330 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.459+0000 7fa6a3fff640 1 -- 172.21.15.100:0/4040266659 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fa6ac0521e0 con 0x7fa6bc0b17d0 2024-03-29T11:42:49.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.459+0000 7fa6a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:49.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.459+0000 7fa6a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:49.463 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:42:49.463 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:49.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.463+0000 7fa6c21a8640 10 librados: watch_flush enter 2024-03-29T11:42:49.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.463+0000 7fa6c21a8640 10 librados: watch_flush exit 2024-03-29T11:42:49.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.463+0000 7fa6c21a8640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:42:49.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.463+0000 7fa6c21a8640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:42:49.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.464+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 >> v1:172.21.15.100:6816/3220289657 conn(0x7fa6a806dda0 legacy=0x7fa6a8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:49.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.464+0000 7fa6c21a8640 10 monclient: shutdown 2024-03-29T11:42:49.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.464+0000 7fa6c21a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:49.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.464+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 >> v1:172.21.15.100:6790/0 conn(0x7fa6bc0b17d0 legacy=0x7fa6bc0ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:49.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.464+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 shutdown_connections 2024-03-29T11:42:49.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.464+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 >> 172.21.15.100:0/4040266659 conn(0x7fa6bc066a00 msgr2=0x7fa6bc13d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:49.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.464+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 shutdown_connections 2024-03-29T11:42:49.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.464+0000 7fa6c21a8640 1 -- 172.21.15.100:0/4040266659 wait complete. 2024-03-29T11:42:49.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:49.465+0000 7fa6c21a8640 1 librados: shutdown 2024-03-29T11:42:49.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:56.291 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:56.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.434+0000 7fdb979c6640 10 monclient: get_monmap_and_config 2024-03-29T11:42:56.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.434+0000 7fdb979c6640 10 monclient: build_initial_monmap 2024-03-29T11:42:56.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.434+0000 7fdb979c6640 10 monclient: monmap: 2024-03-29T11:42:56.439 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:56.439 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:56.439 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:42:56.435794+0000 2024-03-29T11:42:56.439 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:42:56.435794+0000 2024-03-29T11:42:56.439 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:56.440 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:56.440 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:56.440 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:42:56.440 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:42:56.440 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:56.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.435+0000 7fdb979c6640 1 Processor -- start 2024-03-29T11:42:56.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.436+0000 7fdb979c6640 1 -- start start 2024-03-29T11:42:56.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.436+0000 7fdb979c6640 10 monclient: init 2024-03-29T11:42:56.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb979c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:56.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb979c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:56.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb979c6640 10 monclient(hunting): picked mon.noname-c con 0x7fdb9006ade0 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:56.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb979c6640 10 monclient(hunting): picked mon.noname-b con 0x7fdb900afa30 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:56.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb979c6640 10 monclient(hunting): picked mon.noname-a con 0x7fdb901433e0 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:56.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb979c6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb90072670 con 0x7fdb901433e0 2024-03-29T11:42:56.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb979c6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9013da60 con 0x7fdb9006ade0 2024-03-29T11:42:56.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb979c6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb90143fa0 con 0x7fdb900afa30 2024-03-29T11:42:56.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb979c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:56.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb979c6640 10 monclient(hunting): authenticate will time out at 1296.632431s 2024-03-29T11:42:56.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb94f61640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fdb900afa30 0x7fdb9013d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:59064/0 (socket says 172.21.15.100:59064) 2024-03-29T11:42:56.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.437+0000 7fdb94f61640 1 -- 172.21.15.100:0/3922026783 learned_addr learned my addr 172.21.15.100:0/3922026783 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:56.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390688948 0 0) 0x7fdb9013da60 con 0x7fdb9006ade0 2024-03-29T11:42:56.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:56.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:56.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:56.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:42:56.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb7c003520 con 0x7fdb9006ade0 2024-03-29T11:42:56.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310617678 0 0) 0x7fdb90143fa0 con 0x7fdb900afa30 2024-03-29T11:42:56.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:56.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:56.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:56.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:42:56.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb7c0048a0 con 0x7fdb900afa30 2024-03-29T11:42:56.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539081601 0 0) 0x7fdb90072670 con 0x7fdb901433e0 2024-03-29T11:42:56.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:56.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:56.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:56.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:42:56.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.438+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb90143fa0 con 0x7fdb901433e0 2024-03-29T11:42:56.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4249892574 0 0) 0x7fdb7c003520 con 0x7fdb9006ade0 2024-03-29T11:42:56.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 >> v1:172.21.15.100:6790/0 conn(0x7fdb900afa30 legacy=0x7fdb9013d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:56.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 >> v1:172.21.15.100:6789/0 conn(0x7fdb901433e0 legacy=0x7fdb90143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:56.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:56.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 1 monclient: found mon.noname-c 2024-03-29T11:42:56.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:56.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:42:56.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9014f730 con 0x7fdb9006ade0 2024-03-29T11:42:56.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:56.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:56.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:56.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb979c6640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:42:56.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fdb8c0015a0 con 0x7fdb9006ade0 2024-03-29T11:42:56.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb979c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:56.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:56.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:56.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: dump: 2024-03-29T11:42:56.467 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:56.467 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:56.467 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:56.467 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:56.468 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:56.468 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:56.468 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:56.468 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:56.468 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:56.468 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:56.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb979c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:56.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb8c001780 con 0x7fdb9006ade0 2024-03-29T11:42:56.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:56.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 1 -- 172.21.15.100:0/3922026783 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fdb8c001b10 con 0x7fdb9006ade0 2024-03-29T11:42:56.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:56.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:56.470 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb87fff640 10 monclient: dump: 2024-03-29T11:42:56.470 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:56.470 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:56.470 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:56.470 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:56.470 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:56.471 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:56.471 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:56.471 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:56.471 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:56.471 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:56.471 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb979c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:56.471 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.439+0000 7fdb979c6640 4 set_mon_vals no callback set 2024-03-29T11:42:56.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.440+0000 7fdb979c6640 10 monclient: shutdown 2024-03-29T11:42:56.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.440+0000 7fdb979c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:56.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.440+0000 7fdb979c6640 1 -- 172.21.15.100:0/3922026783 >> v1:172.21.15.191:6789/0 conn(0x7fdb9006ade0 legacy=0x7fdb90071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:56.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.440+0000 7fdb979c6640 1 -- 172.21.15.100:0/3922026783 shutdown_connections 2024-03-29T11:42:56.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.440+0000 7fdb979c6640 1 -- 172.21.15.100:0/3922026783 >> 172.21.15.100:0/3922026783 conn(0x7fdb90067470 msgr2=0x7fdb900b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:56.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.441+0000 7fdb979c6640 1 -- 172.21.15.100:0/3922026783 shutdown_connections 2024-03-29T11:42:56.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.441+0000 7fdb979c6640 1 -- 172.21.15.100:0/3922026783 wait complete. 2024-03-29T11:42:56.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.441+0000 7fdb979c6640 10 monclient: build_initial_monmap 2024-03-29T11:42:56.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.441+0000 7fdb979c6640 10 monclient: monmap: 2024-03-29T11:42:56.473 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:42:56.473 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:56.473 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:42:56.473 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:42:56.474 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:56.474 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:56.474 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:42:56.474 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:42:56.474 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:42:56.475 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:56.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.442+0000 7fdb979c6640 1 librados: starting msgr at 2024-03-29T11:42:56.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.442+0000 7fdb979c6640 1 librados: starting objecter 2024-03-29T11:42:56.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.442+0000 7fdb979c6640 1 Processor -- start 2024-03-29T11:42:56.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.442+0000 7fdb979c6640 1 -- start start 2024-03-29T11:42:56.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.442+0000 7fdb979c6640 1 librados: setting wanted keys 2024-03-29T11:42:56.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.442+0000 7fdb979c6640 1 librados: calling monclient init 2024-03-29T11:42:56.476 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.442+0000 7fdb979c6640 10 monclient: init 2024-03-29T11:42:56.476 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb979c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:56.476 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb979c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:56.476 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb979c6640 10 monclient(hunting): picked mon.noname-c con 0x7fdb9006ade0 addr v1:172.21.15.100:6790/0 2024-03-29T11:42:56.476 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb979c6640 10 monclient(hunting): picked mon.noname-a con 0x7fdb900afa30 addr v1:172.21.15.100:6789/0 2024-03-29T11:42:56.476 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb95762640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fdb9006ade0 0x7fdb901db3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:59364/0 (socket says 172.21.15.100:59364) 2024-03-29T11:42:56.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb95762640 1 -- 172.21.15.100:0/2011289196 learned_addr learned my addr 172.21.15.100:0/2011289196 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:42:56.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb979c6640 10 monclient(hunting): picked mon.noname-b con 0x7fdb901433e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:42:56.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9013ff40 con 0x7fdb900afa30 2024-03-29T11:42:56.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb85ffb640 10 client.?.objecter ms_handle_connect 0x7fdb900afa30 2024-03-29T11:42:56.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:56.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb85ffb640 10 client.?.objecter ms_handle_connect 0x7fdb9006ade0 2024-03-29T11:42:56.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb85ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:56.478 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb901daec0 con 0x7fdb901433e0 2024-03-29T11:42:56.478 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb901dab00 con 0x7fdb9006ade0 2024-03-29T11:42:56.478 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb979c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:56.478 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb979c6640 10 monclient(hunting): authenticate will time out at 1296.638902s 2024-03-29T11:42:56.478 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306603793 0 0) 0x7fdb901dab00 con 0x7fdb9006ade0 2024-03-29T11:42:56.478 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:56.479 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:56.479 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:56.479 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb85ffb640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:42:56.479 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.443+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb74004eb0 con 0x7fdb9006ade0 2024-03-29T11:42:56.479 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766345103 0 0) 0x7fdb9013ff40 con 0x7fdb900afa30 2024-03-29T11:42:56.479 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:56.480 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:56.480 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:56.480 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:42:56.480 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb901dab00 con 0x7fdb900afa30 2024-03-29T11:42:56.480 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2097567572 0 0) 0x7fdb74004eb0 con 0x7fdb9006ade0 2024-03-29T11:42:56.480 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb9013ff40 con 0x7fdb9006ade0 2024-03-29T11:42:56.481 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fdb8c001a90 con 0x7fdb9006ade0 2024-03-29T11:42:56.481 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:56.481 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:56.481 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:56.481 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:56.481 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:56.481 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:56.482 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:56.482 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:56.482 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:56.482 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:56.482 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:56.482 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:56.483 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:56.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:56.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 417513177 0 0) 0x7fdb901dab00 con 0x7fdb900afa30 2024-03-29T11:42:56.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb74004eb0 con 0x7fdb900afa30 2024-03-29T11:42:56.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fdb80001970 con 0x7fdb900afa30 2024-03-29T11:42:56.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:56.484 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:56.484 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:56.484 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:56.484 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:56.484 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:56.484 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:56.485 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:56.485 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:56.485 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:56.485 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:56.485 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:56.485 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:56.485 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1026536145 0 0) 0x7fdb9013ff40 con 0x7fdb9006ade0 2024-03-29T11:42:56.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 >> v1:172.21.15.191:6789/0 conn(0x7fdb901433e0 legacy=0x7fdb901e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:56.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 >> v1:172.21.15.100:6789/0 conn(0x7fdb900afa30 legacy=0x7fdb90136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:56.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:56.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 monclient: found mon.c 2024-03-29T11:42:56.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:56.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:56.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb901dac90 con 0x7fdb9006ade0 2024-03-29T11:42:56.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:56.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:56.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:56.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb979c6640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:42:56.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb979c6640 10 monclient: _renew_subs 2024-03-29T11:42:56.488 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb979c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:56.488 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.444+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb900adf70 con 0x7fdb9006ade0 2024-03-29T11:42:56.488 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb979c6640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:56.488 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb979c6640 10 monclient: _renew_subs 2024-03-29T11:42:56.488 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb979c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:56.488 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb90142c50 con 0x7fdb9006ade0 2024-03-29T11:42:56.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb979c6640 1 librados: init done 2024-03-29T11:42:56.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb8c001140 con 0x7fdb9006ade0 2024-03-29T11:42:56.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:56.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fdb8c003a40 con 0x7fdb9006ade0 2024-03-29T11:42:56.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:56.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:56.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb85ffb640 10 monclient: dump: 2024-03-29T11:42:56.490 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:42:56.490 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:42:56.490 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:56.490 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:42:56.490 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:56.490 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:42:56.491 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:42:56.491 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:42:56.491 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:42:56.491 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:42:56.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.445+0000 7fdb86ffd640 4 set_mon_vals no callback set 2024-03-29T11:42:56.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.447+0000 7fdb979c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:56.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.447+0000 7fdb979c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:56.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.447+0000 7fdb979c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:56.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.447+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb70005080 con 0x7fdb9006ade0 2024-03-29T11:42:56.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.448+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fdb8c003ba0 con 0x7fdb9006ade0 2024-03-29T11:42:56.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.449+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fdb8c084b80 con 0x7fdb9006ade0 2024-03-29T11:42:56.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.449+0000 7fdb85ffb640 10 client.4505.objecter ms_dispatch 0x7fdb90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:56.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.449+0000 7fdb85ffb640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:56.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.449+0000 7fdb85ffb640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:56.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.449+0000 7fdb94f61640 10 monclient: get_auth_request con 0x7fdb7406de80 auth_method 0 2024-03-29T11:42:56.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.449+0000 7fdb85ffb640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:42:56.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.450+0000 7fdb85ffb640 10 client.4505.objecter ms_handle_connect 0x7fdb7406de80 2024-03-29T11:42:56.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.459+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb8c050290 con 0x7fdb9006ade0 2024-03-29T11:42:56.495 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.459+0000 7fdb85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:56.495 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.459+0000 7fdb85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:56.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.820+0000 7fdb979c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:56.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.820+0000 7fdb979c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:56.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.820+0000 7fdb979c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:42:56.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.820+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb70005330 con 0x7fdb9006ade0 2024-03-29T11:42:56.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.821+0000 7fdb85ffb640 1 -- 172.21.15.100:0/2011289196 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fdb8c052390 con 0x7fdb9006ade0 2024-03-29T11:42:56.825 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.822+0000 7fdb85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:56.825 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.822+0000 7fdb85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:56.825 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:42:56.825 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:56.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.825+0000 7fdb979c6640 10 librados: watch_flush enter 2024-03-29T11:42:56.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.825+0000 7fdb979c6640 10 librados: watch_flush exit 2024-03-29T11:42:56.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.825+0000 7fdb979c6640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:42:56.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.825+0000 7fdb979c6640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:42:56.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.825+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 >> v1:172.21.15.100:6816/3220289657 conn(0x7fdb7406de80 legacy=0x7fdb74070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:56.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.825+0000 7fdb979c6640 10 monclient: shutdown 2024-03-29T11:42:56.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.825+0000 7fdb979c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:56.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.825+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 >> v1:172.21.15.100:6790/0 conn(0x7fdb9006ade0 legacy=0x7fdb901db3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:56.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.825+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 shutdown_connections 2024-03-29T11:42:56.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.825+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 >> 172.21.15.100:0/2011289196 conn(0x7fdb90067470 msgr2=0x7fdb900b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:56.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.826+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 shutdown_connections 2024-03-29T11:42:56.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.826+0000 7fdb979c6640 1 -- 172.21.15.100:0/2011289196 wait complete. 2024-03-29T11:42:56.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:42:56.826+0000 7fdb979c6640 1 librados: shutdown 2024-03-29T11:42:56.843 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:03.650 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:03.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.799+0000 7fb2faec1640 10 monclient: get_monmap_and_config 2024-03-29T11:43:03.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.799+0000 7fb2faec1640 10 monclient: build_initial_monmap 2024-03-29T11:43:03.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.799+0000 7fb2faec1640 10 monclient: monmap: 2024-03-29T11:43:03.803 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:03.803 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:03.803 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:43:03.800959+0000 2024-03-29T11:43:03.804 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:43:03.800959+0000 2024-03-29T11:43:03.804 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:03.804 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:03.804 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:03.804 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:43:03.804 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:43:03.804 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:03.806 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.802+0000 7fb2faec1640 1 Processor -- start 2024-03-29T11:43:03.806 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.802+0000 7fb2faec1640 1 -- start start 2024-03-29T11:43:03.806 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.802+0000 7fb2faec1640 10 monclient: init 2024-03-29T11:43:03.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.803+0000 7fb2faec1640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:03.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.803+0000 7fb2faec1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:03.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.803+0000 7fb2faec1640 10 monclient(hunting): picked mon.noname-b con 0x7fb2f40b0860 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:03.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.803+0000 7fb2faec1640 10 monclient(hunting): picked mon.noname-a con 0x7fb2f413ef30 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:03.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.803+0000 7fb2faec1640 10 monclient(hunting): picked mon.noname-c con 0x7fb2f40af8e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:03.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.803+0000 7fb2faec1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f40b0c40 con 0x7fb2f413ef30 2024-03-29T11:43:03.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.803+0000 7fb2faec1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f40aed10 con 0x7fb2f40af8e0 2024-03-29T11:43:03.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.803+0000 7fb2faec1640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f40afd50 con 0x7fb2f40b0860 2024-03-29T11:43:03.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.803+0000 7fb2faec1640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:03.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.803+0000 7fb2faec1640 10 monclient(hunting): authenticate will time out at 1303.999029s 2024-03-29T11:43:03.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.806+0000 7fb2f3fff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fb2f413ef30 0x7fb2f40af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57246/0 (socket says 172.21.15.100:57246) 2024-03-29T11:43:03.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.806+0000 7fb2f3fff640 1 -- 172.21.15.100:0/576194603 learned_addr learned my addr 172.21.15.100:0/576194603 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:03.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.806+0000 7fb2f37fe640 1 -- 172.21.15.100:0/576194603 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198078710 0 0) 0x7fb2f40b0c40 con 0x7fb2f413ef30 2024-03-29T11:43:03.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.806+0000 7fb2f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:03.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.806+0000 7fb2f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:03.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.806+0000 7fb2f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:03.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:43:03.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 1 -- 172.21.15.100:0/576194603 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2dc003520 con 0x7fb2f413ef30 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 1 -- 172.21.15.100:0/576194603 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439457612 0 0) 0x7fb2dc003520 con 0x7fb2f413ef30 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 1 -- 172.21.15.100:0/576194603 >> v1:172.21.15.100:6790/0 conn(0x7fb2f40b0860 legacy=0x7fb2f413e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 1 -- 172.21.15.100:0/576194603 >> v1:172.21.15.191:6789/0 conn(0x7fb2f40af8e0 legacy=0x7fb2f4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 1 monclient: found mon.noname-a 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 1 -- 172.21.15.100:0/576194603 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f40ab1b0 con 0x7fb2f413ef30 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 1 -- 172.21.15.100:0/576194603 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb2e4001690 con 0x7fb2f413ef30 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2faec1640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:43:03.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2faec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.807+0000 7fb2f37fe640 10 monclient: dump: 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:03.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.808+0000 7fb2faec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:03.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.808+0000 7fb2f37fe640 1 -- 172.21.15.100:0/576194603 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2e4001860 con 0x7fb2f413ef30 2024-03-29T11:43:03.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.808+0000 7fb2f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:03.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.808+0000 7fb2faec1640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:03.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.808+0000 7fb2faec1640 4 set_mon_vals no callback set 2024-03-29T11:43:03.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2f37fe640 1 -- 172.21.15.100:0/576194603 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb2e4001cb0 con 0x7fb2f413ef30 2024-03-29T11:43:03.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:03.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:03.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2f37fe640 10 monclient: dump: 2024-03-29T11:43:03.814 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:03.814 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:03.814 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2faec1640 10 monclient: shutdown 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2faec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2faec1640 1 -- 172.21.15.100:0/576194603 >> v1:172.21.15.100:6789/0 conn(0x7fb2f413ef30 legacy=0x7fb2f40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2faec1640 1 -- 172.21.15.100:0/576194603 shutdown_connections 2024-03-29T11:43:03.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2faec1640 1 -- 172.21.15.100:0/576194603 >> 172.21.15.100:0/576194603 conn(0x7fb2f406b660 msgr2=0x7fb2f406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:03.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2faec1640 1 -- 172.21.15.100:0/576194603 shutdown_connections 2024-03-29T11:43:03.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.809+0000 7fb2faec1640 1 -- 172.21.15.100:0/576194603 wait complete. 2024-03-29T11:43:03.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.810+0000 7fb2faec1640 10 monclient: build_initial_monmap 2024-03-29T11:43:03.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.810+0000 7fb2faec1640 10 monclient: monmap: 2024-03-29T11:43:03.817 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:03.817 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.811+0000 7fb2faec1640 1 librados: starting msgr at 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.811+0000 7fb2faec1640 1 librados: starting objecter 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.811+0000 7fb2faec1640 1 Processor -- start 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.811+0000 7fb2faec1640 1 -- start start 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.811+0000 7fb2faec1640 1 librados: setting wanted keys 2024-03-29T11:43:03.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.811+0000 7fb2faec1640 1 librados: calling monclient init 2024-03-29T11:43:03.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.811+0000 7fb2faec1640 10 monclient: init 2024-03-29T11:43:03.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.812+0000 7fb2faec1640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:03.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.812+0000 7fb2faec1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:03.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.812+0000 7fb2faec1640 10 monclient(hunting): picked mon.noname-a con 0x7fb2f40af8e0 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:03.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.812+0000 7fb2faec1640 10 monclient(hunting): picked mon.noname-b con 0x7fb2f40b0860 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:03.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.812+0000 7fb2faec1640 10 monclient(hunting): picked mon.noname-c con 0x7fb2f413ef30 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:03.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.812+0000 7fb2faec1640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f4136b80 con 0x7fb2f40af8e0 2024-03-29T11:43:03.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.812+0000 7fb2faec1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f40b1830 con 0x7fb2f40b0860 2024-03-29T11:43:03.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.812+0000 7fb2faec1640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f40b1b10 con 0x7fb2f413ef30 2024-03-29T11:43:03.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.812+0000 7fb2faec1640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:03.820 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.812+0000 7fb2faec1640 10 monclient(hunting): authenticate will time out at 1304.007490s 2024-03-29T11:43:03.822 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.818+0000 7fb2f945e640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fb2f413ef30 0x7fb2f4135c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:50063/0 (socket says 172.21.15.100:50063) 2024-03-29T11:43:03.822 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.818+0000 7fb2f945e640 1 -- 172.21.15.100:0/1769494470 learned_addr learned my addr 172.21.15.100:0/1769494470 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:03.822 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.818+0000 7fb2f17fa640 10 client.?.objecter ms_handle_connect 0x7fb2f413ef30 2024-03-29T11:43:03.822 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.818+0000 7fb2f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:03.822 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924144967 0 0) 0x7fb2f40b1b10 con 0x7fb2f413ef30 2024-03-29T11:43:03.822 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e0004eb0 con 0x7fb2f413ef30 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 client.?.objecter ms_handle_connect 0x7fb2f40af8e0 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303252168 0 0) 0x7fb2f4136b80 con 0x7fb2f40af8e0 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:03.823 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:43:03.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2f40b1b10 con 0x7fb2f40af8e0 2024-03-29T11:43:03.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 client.?.objecter ms_handle_connect 0x7fb2f40b0860 2024-03-29T11:43:03.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.819+0000 7fb2f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:03.826 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725721252 0 0) 0x7fb2f40b1830 con 0x7fb2f40b0860 2024-03-29T11:43:03.826 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:03.826 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:03.826 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:03.826 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:43:03.826 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2f4136b80 con 0x7fb2f40b0860 2024-03-29T11:43:03.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 964996681 0 0) 0x7fb2e0004eb0 con 0x7fb2f413ef30 2024-03-29T11:43:03.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2f40b1830 con 0x7fb2f413ef30 2024-03-29T11:43:03.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb2e80019b0 con 0x7fb2f413ef30 2024-03-29T11:43:03.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:03.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:03.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient(hunting): dump: 2024-03-29T11:43:03.827 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:03.827 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:03.827 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:03.828 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:03.828 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:03.828 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:03.828 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:03.828 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:03.828 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:03.828 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:03.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:03.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1630020922 0 0) 0x7fb2f40b1830 con 0x7fb2f413ef30 2024-03-29T11:43:03.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 >> v1:172.21.15.191:6789/0 conn(0x7fb2f40b0860 legacy=0x7fb2f4135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 >> v1:172.21.15.100:6789/0 conn(0x7fb2f40af8e0 legacy=0x7fb2f40b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 1 monclient: found mon.c 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f40b2640 con 0x7fb2f413ef30 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2faec1640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2faec1640 10 monclient: _renew_subs 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2faec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2faec1640 1 -- 172.21.15.100:0/1769494470 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f41363a0 con 0x7fb2f413ef30 2024-03-29T11:43:03.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2faec1640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2faec1640 10 monclient: _renew_subs 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2faec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2faec1640 1 -- 172.21.15.100:0/1769494470 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f4137cd0 con 0x7fb2f413ef30 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.820+0000 7fb2faec1640 1 librados: init done 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.821+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2e8003610 con 0x7fb2f413ef30 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.821+0000 7fb2f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.821+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fb2e80039a0 con 0x7fb2f413ef30 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.821+0000 7fb2f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.821+0000 7fb2f2ffd640 4 set_mon_vals no callback set 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.821+0000 7fb2f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.821+0000 7fb2f17fa640 10 monclient: dump: 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:03.830 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:03.831 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:03.831 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:03.831 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:03.831 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:03.831 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:03.831 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:03.831 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:03.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.824+0000 7fb2faec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:03.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.824+0000 7fb2faec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:03.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.824+0000 7fb2faec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:03.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.824+0000 7fb2faec1640 1 -- 172.21.15.100:0/1769494470 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d4005080 con 0x7fb2f413ef30 2024-03-29T11:43:03.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.825+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fb2e8003b00 con 0x7fb2f413ef30 2024-03-29T11:43:03.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.825+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fb2e8084b20 con 0x7fb2f413ef30 2024-03-29T11:43:03.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.826+0000 7fb2f8c5d640 10 monclient: get_auth_request con 0x7fb2e006dbe0 auth_method 0 2024-03-29T11:43:03.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.826+0000 7fb2f17fa640 10 client.4508.objecter ms_dispatch 0x7fb2f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:03.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.826+0000 7fb2f17fa640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:03.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.826+0000 7fb2f17fa640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:03.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.826+0000 7fb2f17fa640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T11:43:03.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.826+0000 7fb2f17fa640 10 client.4508.objecter ms_handle_connect 0x7fb2e006dbe0 2024-03-29T11:43:03.836 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.833+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2e8050400 con 0x7fb2f413ef30 2024-03-29T11:43:03.836 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.833+0000 7fb2f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:03.836 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:03.833+0000 7fb2f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:04.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.161+0000 7fb2faec1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:04.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.161+0000 7fb2faec1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:04.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.161+0000 7fb2faec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:04.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.161+0000 7fb2faec1640 1 -- 172.21.15.100:0/1769494470 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2d4005330 con 0x7fb2f413ef30 2024-03-29T11:43:04.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.162+0000 7fb2f17fa640 1 -- 172.21.15.100:0/1769494470 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fb2e8052330 con 0x7fb2f413ef30 2024-03-29T11:43:04.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.162+0000 7fb2f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:04.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.162+0000 7fb2f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:04.167 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:43:04.168 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:04.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.167+0000 7fb2faec1640 10 librados: watch_flush enter 2024-03-29T11:43:04.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.167+0000 7fb2faec1640 10 librados: watch_flush exit 2024-03-29T11:43:04.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.167+0000 7fb2faec1640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T11:43:04.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.167+0000 7fb2faec1640 10 client.4508.objecter successfully canceled tick 2024-03-29T11:43:04.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.167+0000 7fb2faec1640 1 -- 172.21.15.100:0/1769494470 >> v1:172.21.15.100:6816/3220289657 conn(0x7fb2e006dbe0 legacy=0x7fb2e00700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:04.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.167+0000 7fb2faec1640 10 monclient: shutdown 2024-03-29T11:43:04.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.167+0000 7fb2faec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:04.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.167+0000 7fb2faec1640 1 -- 172.21.15.100:0/1769494470 >> v1:172.21.15.100:6790/0 conn(0x7fb2f413ef30 legacy=0x7fb2f4135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:04.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.168+0000 7fb2faec1640 1 -- 172.21.15.100:0/1769494470 shutdown_connections 2024-03-29T11:43:04.172 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.168+0000 7fb2faec1640 1 -- 172.21.15.100:0/1769494470 >> 172.21.15.100:0/1769494470 conn(0x7fb2f406b660 msgr2=0x7fb2f40ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:04.172 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.168+0000 7fb2faec1640 1 -- 172.21.15.100:0/1769494470 shutdown_connections 2024-03-29T11:43:04.172 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.168+0000 7fb2faec1640 1 -- 172.21.15.100:0/1769494470 wait complete. 2024-03-29T11:43:04.172 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:04.169+0000 7fb2faec1640 1 librados: shutdown 2024-03-29T11:43:04.189 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:10.997 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:11.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.144+0000 7ff89b94b640 10 monclient: get_monmap_and_config 2024-03-29T11:43:11.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.144+0000 7ff89b94b640 10 monclient: build_initial_monmap 2024-03-29T11:43:11.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.144+0000 7ff89b94b640 10 monclient: monmap: 2024-03-29T11:43:11.148 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:11.148 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:11.148 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:43:11.145570+0000 2024-03-29T11:43:11.148 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:43:11.145570+0000 2024-03-29T11:43:11.148 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:11.148 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:11.148 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:11.149 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:43:11.149 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:43:11.149 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:11.151 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.148+0000 7ff89b94b640 1 Processor -- start 2024-03-29T11:43:11.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.148+0000 7ff89b94b640 1 -- start start 2024-03-29T11:43:11.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.148+0000 7ff89b94b640 10 monclient: init 2024-03-29T11:43:11.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.149+0000 7ff89b94b640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:11.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.149+0000 7ff89b94b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:11.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.149+0000 7ff89b94b640 10 monclient(hunting): picked mon.noname-c con 0x7ff8940b1f60 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:11.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.149+0000 7ff89b94b640 10 monclient(hunting): picked mon.noname-a con 0x7ff894071eb0 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:11.156 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.149+0000 7ff89b94b640 10 monclient(hunting): picked mon.noname-b con 0x7ff894143d10 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:11.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.150+0000 7ff898ee6640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff894071eb0 0x7ff894146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:55040/0 (socket says 172.21.15.100:55040) 2024-03-29T11:43:11.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.153+0000 7ff898ee6640 1 -- 172.21.15.100:0/760785589 learned_addr learned my addr 172.21.15.100:0/760785589 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:11.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.153+0000 7ff89b94b640 1 -- 172.21.15.100:0/760785589 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89406ef80 con 0x7ff894071eb0 2024-03-29T11:43:11.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.153+0000 7ff89b94b640 1 -- 172.21.15.100:0/760785589 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89406f260 con 0x7ff8940b1f60 2024-03-29T11:43:11.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.153+0000 7ff89b94b640 1 -- 172.21.15.100:0/760785589 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8940b40c0 con 0x7ff894143d10 2024-03-29T11:43:11.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.153+0000 7ff89b94b640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:11.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.153+0000 7ff89b94b640 10 monclient(hunting): authenticate will time out at 1311.348790s 2024-03-29T11:43:11.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172084386 0 0) 0x7ff89406ef80 con 0x7ff894071eb0 2024-03-29T11:43:11.157 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:11.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:11.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:11.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:43:11.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff880003520 con 0x7ff894071eb0 2024-03-29T11:43:11.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875263461 0 0) 0x7ff8940b40c0 con 0x7ff894143d10 2024-03-29T11:43:11.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:11.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:11.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:11.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:43:11.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89406ef80 con 0x7ff894143d10 2024-03-29T11:43:11.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4124012915 0 0) 0x7ff880003520 con 0x7ff894071eb0 2024-03-29T11:43:11.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 >> v1:172.21.15.100:6790/0 conn(0x7ff894143d10 legacy=0x7ff894146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:11.159 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 >> v1:172.21.15.191:6789/0 conn(0x7ff8940b1f60 legacy=0x7ff894143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 1 monclient: found mon.noname-a 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff894154d60 con 0x7ff894071eb0 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient: _finish_auth 0 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff890001690 con 0x7ff894071eb0 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.154+0000 7ff88bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.155+0000 7ff88bfff640 10 monclient: dump: 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:11.160 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.155+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8900018b0 con 0x7ff894071eb0 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.155+0000 7ff88bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.155+0000 7ff88bfff640 1 -- 172.21.15.100:0/760785589 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff890001c40 con 0x7ff894071eb0 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.155+0000 7ff88bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.155+0000 7ff88bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.155+0000 7ff88bfff640 10 monclient: dump: 2024-03-29T11:43:11.161 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.156+0000 7ff89b94b640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.156+0000 7ff89b94b640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:11.162 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.156+0000 7ff89b94b640 4 set_mon_vals no callback set 2024-03-29T11:43:11.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.156+0000 7ff89b94b640 10 monclient: shutdown 2024-03-29T11:43:11.163 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.156+0000 7ff89b94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.156+0000 7ff89b94b640 1 -- 172.21.15.100:0/760785589 >> v1:172.21.15.100:6789/0 conn(0x7ff894071eb0 legacy=0x7ff894146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.157+0000 7ff89b94b640 1 -- 172.21.15.100:0/760785589 shutdown_connections 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.157+0000 7ff89b94b640 1 -- 172.21.15.100:0/760785589 >> 172.21.15.100:0/760785589 conn(0x7ff894066a00 msgr2=0x7ff894066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.159+0000 7ff89b94b640 1 -- 172.21.15.100:0/760785589 shutdown_connections 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.159+0000 7ff89b94b640 1 -- 172.21.15.100:0/760785589 wait complete. 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.160+0000 7ff89b94b640 10 monclient: build_initial_monmap 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.160+0000 7ff89b94b640 10 monclient: monmap: 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:11.164 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:11.165 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:43:11.165 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:43:11.165 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:11.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.160+0000 7ff89b94b640 1 librados: starting msgr at 2024-03-29T11:43:11.165 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.160+0000 7ff89b94b640 1 librados: starting objecter 2024-03-29T11:43:11.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.160+0000 7ff89b94b640 1 Processor -- start 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.161+0000 7ff89b94b640 1 -- start start 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.161+0000 7ff89b94b640 1 librados: setting wanted keys 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.161+0000 7ff89b94b640 1 librados: calling monclient init 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.161+0000 7ff89b94b640 10 monclient: init 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.162+0000 7ff89b94b640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.162+0000 7ff89b94b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.162+0000 7ff89b94b640 10 monclient(hunting): picked mon.noname-b con 0x7ff8940b1f60 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.162+0000 7ff89b94b640 10 monclient(hunting): picked mon.noname-a con 0x7ff894143d10 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.162+0000 7ff89b94b640 10 monclient(hunting): picked mon.noname-c con 0x7ff8941386e0 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.162+0000 7ff89b94b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8941da310 con 0x7ff894143d10 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.162+0000 7ff89b94b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff894134a90 con 0x7ff8940b1f60 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.162+0000 7ff89b94b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff894071310 con 0x7ff8941386e0 2024-03-29T11:43:11.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.162+0000 7ff89b94b640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:11.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.162+0000 7ff89b94b640 10 monclient(hunting): authenticate will time out at 1311.357665s 2024-03-29T11:43:11.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.165+0000 7ff8996e7640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ff8940b1f60 0x7ff8941d9c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:35959/0 (socket says 172.21.15.100:35959) 2024-03-29T11:43:11.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.165+0000 7ff898ee6640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff894143d10 0x7ff894133c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:56000/0 (socket says 172.21.15.100:56000) 2024-03-29T11:43:11.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.165+0000 7ff898ee6640 1 -- 172.21.15.100:0/2785791451 learned_addr learned my addr 172.21.15.100:0/2785791451 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.167+0000 7ff889ffb640 10 client.?.objecter ms_handle_connect 0x7ff894143d10 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.167+0000 7ff889ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.167+0000 7ff889ffb640 10 client.?.objecter ms_handle_connect 0x7ff8941386e0 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.167+0000 7ff889ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.167+0000 7ff889ffb640 10 client.?.objecter ms_handle_connect 0x7ff8940b1f60 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.167+0000 7ff889ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.167+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741442966 0 0) 0x7ff8941da310 con 0x7ff894143d10 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.167+0000 7ff889ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff878004eb0 con 0x7ff894143d10 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466150925 0 0) 0x7ff894071310 con 0x7ff8941386e0 2024-03-29T11:43:11.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8941da310 con 0x7ff8941386e0 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974383095 0 0) 0x7ff894134a90 con 0x7ff8940b1f60 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.168+0000 7ff889ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.169+0000 7ff889ffb640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.169+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff894071310 con 0x7ff8940b1f60 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.169+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035345523 0 0) 0x7ff878004eb0 con 0x7ff894143d10 2024-03-29T11:43:11.174 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.169+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff894134a90 con 0x7ff894143d10 2024-03-29T11:43:11.175 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff890001690 con 0x7ff894143d10 2024-03-29T11:43:11.175 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:11.175 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:11.175 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 10 monclient(hunting): dump: 2024-03-29T11:43:11.175 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:11.175 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:11.175 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:11.175 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:11.175 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 895934441 0 0) 0x7ff8941da310 con 0x7ff8941386e0 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff878004eb0 con 0x7ff8941386e0 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff884001700 con 0x7ff8941386e0 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.170+0000 7ff889ffb640 10 monclient(hunting): dump: 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:11.176 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3938572898 0 0) 0x7ff878004eb0 con 0x7ff8941386e0 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 >> v1:172.21.15.191:6789/0 conn(0x7ff8940b1f60 legacy=0x7ff8941d9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 >> v1:172.21.15.100:6789/0 conn(0x7ff894143d10 legacy=0x7ff894133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 1 monclient: found mon.c 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:11.177 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89413a2d0 con 0x7ff8941386e0 2024-03-29T11:43:11.178 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:11.178 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:11.178 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:11.179 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff884001bb0 con 0x7ff8941386e0 2024-03-29T11:43:11.179 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff889ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:11.179 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.171+0000 7ff88affd640 4 set_mon_vals no callback set 2024-03-29T11:43:11.179 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.172+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff884003790 con 0x7ff8941386e0 2024-03-29T11:43:11.179 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.172+0000 7ff889ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.172+0000 7ff889ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.172+0000 7ff889ffb640 10 monclient: dump: 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.174+0000 7ff89b94b640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:43:11.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.174+0000 7ff89b94b640 10 monclient: _renew_subs 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.174+0000 7ff89b94b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.174+0000 7ff89b94b640 1 -- 172.21.15.100:0/2785791451 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89413a5b0 con 0x7ff8941386e0 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.174+0000 7ff89b94b640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.174+0000 7ff89b94b640 10 monclient: _renew_subs 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.174+0000 7ff89b94b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.174+0000 7ff89b94b640 1 -- 172.21.15.100:0/2785791451 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff894070f10 con 0x7ff8941386e0 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.174+0000 7ff89b94b640 1 librados: init done 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.176+0000 7ff89b94b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.176+0000 7ff89b94b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.176+0000 7ff89b94b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:11.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.176+0000 7ff89b94b640 1 -- 172.21.15.100:0/2785791451 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff894075f10 con 0x7ff8941386e0 2024-03-29T11:43:11.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.179+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7ff884019760 con 0x7ff8941386e0 2024-03-29T11:43:11.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.180+0000 7ff8996e7640 10 monclient: get_auth_request con 0x7ff87806de80 auth_method 0 2024-03-29T11:43:11.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.181+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7ff8840834d0 con 0x7ff8941386e0 2024-03-29T11:43:11.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.181+0000 7ff889ffb640 10 client.4514.objecter ms_dispatch 0x7ff894000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:11.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.181+0000 7ff889ffb640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:11.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.181+0000 7ff889ffb640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:11.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.181+0000 7ff889ffb640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T11:43:11.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.181+0000 7ff889ffb640 10 client.4514.objecter ms_handle_connect 0x7ff87806de80 2024-03-29T11:43:11.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.185+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff88404edb0 con 0x7ff8941386e0 2024-03-29T11:43:11.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.185+0000 7ff889ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:11.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.185+0000 7ff889ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:11.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.510+0000 7ff89b94b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:11.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.510+0000 7ff89b94b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:11.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.510+0000 7ff89b94b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:11.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.510+0000 7ff89b94b640 1 -- 172.21.15.100:0/2785791451 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff89406f700 con 0x7ff8941386e0 2024-03-29T11:43:11.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.511+0000 7ff889ffb640 1 -- 172.21.15.100:0/2785791451 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7ff89406f700 con 0x7ff8941386e0 2024-03-29T11:43:11.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.511+0000 7ff889ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:11.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.511+0000 7ff889ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:11.515 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:43:11.515 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:11.517 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.514+0000 7ff89b94b640 10 librados: watch_flush enter 2024-03-29T11:43:11.518 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.515+0000 7ff89b94b640 10 librados: watch_flush exit 2024-03-29T11:43:11.518 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.515+0000 7ff89b94b640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T11:43:11.518 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.515+0000 7ff89b94b640 10 client.4514.objecter successfully canceled tick 2024-03-29T11:43:11.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.515+0000 7ff89b94b640 1 -- 172.21.15.100:0/2785791451 >> v1:172.21.15.100:6816/3220289657 conn(0x7ff87806de80 legacy=0x7ff878070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:11.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.515+0000 7ff89b94b640 10 monclient: shutdown 2024-03-29T11:43:11.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.515+0000 7ff89b94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:11.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.515+0000 7ff89b94b640 1 -- 172.21.15.100:0/2785791451 >> v1:172.21.15.100:6790/0 conn(0x7ff8941386e0 legacy=0x7ff894134380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:11.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.515+0000 7ff89b94b640 1 -- 172.21.15.100:0/2785791451 shutdown_connections 2024-03-29T11:43:11.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.515+0000 7ff89b94b640 1 -- 172.21.15.100:0/2785791451 >> 172.21.15.100:0/2785791451 conn(0x7ff894066a00 msgr2=0x7ff8940b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:11.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.516+0000 7ff89b94b640 1 -- 172.21.15.100:0/2785791451 shutdown_connections 2024-03-29T11:43:11.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.516+0000 7ff89b94b640 1 -- 172.21.15.100:0/2785791451 wait complete. 2024-03-29T11:43:11.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:11.516+0000 7ff89b94b640 1 librados: shutdown 2024-03-29T11:43:11.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:18.344 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:18.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.485+0000 7f74aad74640 10 monclient: get_monmap_and_config 2024-03-29T11:43:18.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.485+0000 7f74aad74640 10 monclient: build_initial_monmap 2024-03-29T11:43:18.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.485+0000 7f74aad74640 10 monclient: monmap: 2024-03-29T11:43:18.490 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:18.490 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:18.490 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:43:18.486868+0000 2024-03-29T11:43:18.490 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:43:18.486868+0000 2024-03-29T11:43:18.490 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:18.490 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:18.490 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:18.490 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:43:18.490 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:43:18.490 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:18.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.486+0000 7f74aad74640 1 Processor -- start 2024-03-29T11:43:18.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.487+0000 7f74aad74640 1 -- start start 2024-03-29T11:43:18.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.487+0000 7f74aad74640 10 monclient: init 2024-03-29T11:43:18.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.488+0000 7f74aad74640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:18.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.488+0000 7f74aad74640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:18.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.488+0000 7f74aad74640 10 monclient(hunting): picked mon.noname-c con 0x7f74a40a9750 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:18.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.488+0000 7f74aad74640 10 monclient(hunting): picked mon.noname-a con 0x7f74a40add40 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:18.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.488+0000 7f74aad74640 10 monclient(hunting): picked mon.noname-b con 0x7f74a40aad20 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:18.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.488+0000 7f74aad74640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a4149870 con 0x7f74a40add40 2024-03-29T11:43:18.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.488+0000 7f74aad74640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a40abf80 con 0x7f74a40a9750 2024-03-29T11:43:18.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.488+0000 7f74aad74640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a4149ac0 con 0x7f74a40aad20 2024-03-29T11:43:18.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.488+0000 7f74aad74640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:18.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.488+0000 7f74aad74640 10 monclient(hunting): authenticate will time out at 1318.683764s 2024-03-29T11:43:18.495 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.491+0000 7f74aa573640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f74a40aad20 0x7f74a4151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:45532/0 (socket says 172.21.15.100:45532) 2024-03-29T11:43:18.495 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.491+0000 7f74aa573640 1 -- 172.21.15.100:0/1004987552 learned_addr learned my addr 172.21.15.100:0/1004987552 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:18.497 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.492+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691148967 0 0) 0x7f74a4149ac0 con 0x7f74a40aad20 2024-03-29T11:43:18.497 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.492+0000 7f74a8d70640 10 monclient(hunting): _negotiate 2024-03-29T11:43:18.497 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.492+0000 7f74a8d70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:18.497 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.492+0000 7f74a8d70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:18.497 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.492+0000 7f74a8d70640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:43:18.498 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.492+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7490003520 con 0x7f74a40aad20 2024-03-29T11:43:18.498 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.493+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063220130 0 0) 0x7f74a40abf80 con 0x7f74a40a9750 2024-03-29T11:43:18.498 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.493+0000 7f74a8d70640 10 monclient(hunting): _negotiate 2024-03-29T11:43:18.498 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.493+0000 7f74a8d70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:18.498 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.493+0000 7f74a8d70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:18.498 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.493+0000 7f74a8d70640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:43:18.498 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.493+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a4149ac0 con 0x7f74a40a9750 2024-03-29T11:43:18.498 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.493+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3482375146 0 0) 0x7f7490003520 con 0x7f74a40aad20 2024-03-29T11:43:18.498 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.493+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 >> v1:172.21.15.191:6789/0 conn(0x7f74a40a9750 legacy=0x7f74a4149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:18.499 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.493+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 >> v1:172.21.15.100:6789/0 conn(0x7f74a40add40 legacy=0x7f74a4143610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:18.502 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 10 monclient: _finish_hunting 0 2024-03-29T11:43:18.502 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 1 monclient: found mon.noname-b 2024-03-29T11:43:18.502 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:18.502 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:43:18.502 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a414a6f0 con 0x7f74a40aad20 2024-03-29T11:43:18.503 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 10 monclient: _finish_auth 0 2024-03-29T11:43:18.503 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 10 monclient: _check_auth_tickets 2024-03-29T11:43:18.503 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:18.503 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74aad74640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:43:18.503 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74aad74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:18.503 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f7494001690 con 0x7f74a40aad20 2024-03-29T11:43:18.503 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:18.503 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:18.503 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74a8d70640 10 monclient: dump: 2024-03-29T11:43:18.504 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:18.504 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:18.504 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:18.504 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:18.504 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:18.504 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:18.504 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:18.504 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:18.504 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:18.504 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.495+0000 7f74aad74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.496+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74940017f0 con 0x7f74a40aad20 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.496+0000 7f74a8d70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.496+0000 7f74aad74640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.496+0000 7f74aad74640 4 set_mon_vals no callback set 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.496+0000 7f74a8d70640 1 -- 172.21.15.100:0/1004987552 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f7494001c00 con 0x7f74a40aad20 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.496+0000 7f74a8d70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.496+0000 7f74a8d70640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.496+0000 7f74a8d70640 10 monclient: dump: 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:18.505 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.497+0000 7f74aad74640 10 monclient: shutdown 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.497+0000 7f74aad74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.497+0000 7f74aad74640 1 -- 172.21.15.100:0/1004987552 >> v1:172.21.15.100:6790/0 conn(0x7f74a40aad20 legacy=0x7f74a4151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.497+0000 7f74aad74640 1 -- 172.21.15.100:0/1004987552 shutdown_connections 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.497+0000 7f74aad74640 1 -- 172.21.15.100:0/1004987552 >> 172.21.15.100:0/1004987552 conn(0x7f74a40673e0 msgr2=0x7f74a40b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:18.506 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.498+0000 7f74aad74640 1 -- 172.21.15.100:0/1004987552 shutdown_connections 2024-03-29T11:43:18.507 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.498+0000 7f74aad74640 1 -- 172.21.15.100:0/1004987552 wait complete. 2024-03-29T11:43:18.507 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.499+0000 7f74aad74640 10 monclient: build_initial_monmap 2024-03-29T11:43:18.507 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.499+0000 7f74aad74640 10 monclient: monmap: 2024-03-29T11:43:18.507 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.500+0000 7f74aad74640 1 librados: starting msgr at 2024-03-29T11:43:18.508 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.500+0000 7f74aad74640 1 librados: starting objecter 2024-03-29T11:43:18.509 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.500+0000 7f74aad74640 1 Processor -- start 2024-03-29T11:43:18.509 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.501+0000 7f74aad74640 1 -- start start 2024-03-29T11:43:18.509 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.501+0000 7f74aad74640 1 librados: setting wanted keys 2024-03-29T11:43:18.509 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.501+0000 7f74aad74640 1 librados: calling monclient init 2024-03-29T11:43:18.509 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.501+0000 7f74aad74640 10 monclient: init 2024-03-29T11:43:18.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.502+0000 7f74aad74640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:18.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.502+0000 7f74aad74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:18.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.502+0000 7f74aad74640 10 monclient(hunting): picked mon.noname-a con 0x7f74a40a9750 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:18.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.502+0000 7f74aad74640 10 monclient(hunting): picked mon.noname-c con 0x7f74a40aad20 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:18.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.502+0000 7f74aad74640 10 monclient(hunting): picked mon.noname-b con 0x7f74a40add40 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:18.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.502+0000 7f74aad74640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a4070fd0 con 0x7f74a40a9750 2024-03-29T11:43:18.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.502+0000 7f74aad74640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a40708d0 con 0x7f74a40add40 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.502+0000 7f74aad74640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a4070bb0 con 0x7f74a40aad20 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.502+0000 7f74aad74640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.502+0000 7f74aad74640 10 monclient(hunting): authenticate will time out at 1318.697771s 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.505+0000 7f74a9571640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f74a40aad20 0x7f74a4137670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:49612/0 (socket says 172.21.15.100:49612) 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.505+0000 7f74a9571640 1 -- 172.21.15.100:0/3167333279 learned_addr learned my addr 172.21.15.100:0/3167333279 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.505+0000 7f749a7fc640 10 client.?.objecter ms_handle_connect 0x7f74a40aad20 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.506+0000 7f749a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.506+0000 7f749a7fc640 10 client.?.objecter ms_handle_connect 0x7f74a40add40 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.506+0000 7f749a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.506+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521042508 0 0) 0x7f74a40708d0 con 0x7f74a40add40 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.506+0000 7f749a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.506+0000 7f749a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.506+0000 7f749a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:18.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.506+0000 7f749a7fc640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.506+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7488004eb0 con 0x7f74a40add40 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487361619 0 0) 0x7f74a4070bb0 con 0x7f74a40aad20 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a40708d0 con 0x7f74a40aad20 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 783586207 0 0) 0x7f7488004eb0 con 0x7f74a40add40 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74a4070bb0 con 0x7f74a40add40 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f7494001b00 con 0x7f74a40add40 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:18.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): dump: 2024-03-29T11:43:18.514 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:18.514 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:18.514 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:18.514 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:18.514 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:18.514 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:18.514 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:18.514 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2612212885 0 0) 0x7f74a40708d0 con 0x7f74a40aad20 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7488004eb0 con 0x7f74a40aad20 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f74a00017d0 con 0x7f74a40aad20 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 10 monclient(hunting): dump: 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:18.515 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:18.516 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:18.516 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:18.516 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:18.516 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:18.516 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:18.516 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 168022904 0 0) 0x7f74a4070bb0 con 0x7f74a40add40 2024-03-29T11:43:18.516 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 >> v1:172.21.15.100:6790/0 conn(0x7f74a40aad20 legacy=0x7f74a4137670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:18.516 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.507+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 >> v1:172.21.15.100:6789/0 conn(0x7f74a40a9750 legacy=0x7f74a406efc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:18.518 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.509+0000 7f749a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.509+0000 7f749a7fc640 1 monclient: found mon.b 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.509+0000 7f749a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.509+0000 7f749a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.509+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a406fa40 con 0x7f74a40add40 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.509+0000 7f749a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.509+0000 7f749a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f749a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f74aad74640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f74aad74640 10 monclient: _renew_subs 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f74aad74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f74aad74640 1 -- 172.21.15.100:0/3167333279 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a406e550 con 0x7f74a40add40 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f74aad74640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:18.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f74aad74640 10 monclient: _renew_subs 2024-03-29T11:43:18.520 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f74aad74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:43:18.520 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f74aad74640 1 -- 172.21.15.100:0/3167333279 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a4137d80 con 0x7f74a40add40 2024-03-29T11:43:18.520 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f74aad74640 1 librados: init done 2024-03-29T11:43:18.520 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7494004a90 con 0x7f74a40add40 2024-03-29T11:43:18.520 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f749a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:18.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f7494004da0 con 0x7f74a40add40 2024-03-29T11:43:18.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f749a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:18.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f749a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:18.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f749a7fc640 10 monclient: dump: 2024-03-29T11:43:18.525 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:18.525 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:18.525 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:18.525 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.510+0000 7f749bfff640 4 set_mon_vals no callback set 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.511+0000 7f74aad74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.511+0000 7f74aad74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.511+0000 7f74aad74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.511+0000 7f74aad74640 1 -- 172.21.15.100:0/3167333279 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748c005080 con 0x7f74a40add40 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.513+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f749401ad70 con 0x7f74a40add40 2024-03-29T11:43:18.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.514+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f7494084a00 con 0x7f74a40add40 2024-03-29T11:43:18.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.514+0000 7f749a7fc640 10 client.4423.objecter ms_dispatch 0x7f74a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:18.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.514+0000 7f749a7fc640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:18.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.514+0000 7f749a7fc640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:18.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.514+0000 7f749a7fc640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:43:18.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.514+0000 7f74a9d72640 10 monclient: get_auth_request con 0x7f748806de30 auth_method 0 2024-03-29T11:43:18.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.515+0000 7f749a7fc640 10 client.4423.objecter ms_handle_connect 0x7f748806de30 2024-03-29T11:43:18.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.519+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74940502e0 con 0x7f74a40add40 2024-03-29T11:43:18.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.519+0000 7f749a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:18.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.519+0000 7f749a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:18.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.849+0000 7f74aad74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:18.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.849+0000 7f74aad74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:18.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.849+0000 7f74aad74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:43:18.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.849+0000 7f74aad74640 1 -- 172.21.15.100:0/3167333279 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f748c005330 con 0x7f74a40add40 2024-03-29T11:43:18.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.850+0000 7f749a7fc640 1 -- 172.21.15.100:0/3167333279 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f7494052210 con 0x7f74a40add40 2024-03-29T11:43:18.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.850+0000 7f749a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:18.854 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.850+0000 7f749a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:18.854 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:43:18.854 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:18.858 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.855+0000 7f74aad74640 10 librados: watch_flush enter 2024-03-29T11:43:18.862 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.855+0000 7f74aad74640 10 librados: watch_flush exit 2024-03-29T11:43:18.862 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.855+0000 7f74aad74640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:43:18.862 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.855+0000 7f74aad74640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:43:18.862 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.855+0000 7f74aad74640 1 -- 172.21.15.100:0/3167333279 >> v1:172.21.15.100:6816/3220289657 conn(0x7f748806de30 legacy=0x7f74880702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:18.862 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.855+0000 7f74aad74640 10 monclient: shutdown 2024-03-29T11:43:18.862 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.855+0000 7f74aad74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:18.862 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.855+0000 7f74aad74640 1 -- 172.21.15.100:0/3167333279 >> v1:172.21.15.191:6789/0 conn(0x7f74a40add40 legacy=0x7f74a4132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:18.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.859+0000 7f74aad74640 1 -- 172.21.15.100:0/3167333279 shutdown_connections 2024-03-29T11:43:18.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.859+0000 7f74aad74640 1 -- 172.21.15.100:0/3167333279 >> 172.21.15.100:0/3167333279 conn(0x7f74a40673e0 msgr2=0x7f74a40aa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:18.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.859+0000 7f74aad74640 1 -- 172.21.15.100:0/3167333279 shutdown_connections 2024-03-29T11:43:18.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.859+0000 7f74aad74640 1 -- 172.21.15.100:0/3167333279 wait complete. 2024-03-29T11:43:18.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:18.859+0000 7f74aad74640 1 librados: shutdown 2024-03-29T11:43:18.881 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:25.688 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:25.838 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.835+0000 7f630ed80640 10 monclient: get_monmap_and_config 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.835+0000 7f630ed80640 10 monclient: build_initial_monmap 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.835+0000 7f630ed80640 10 monclient: monmap: 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:43:25.836432+0000 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:43:25.836432+0000 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:43:25.839 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:25.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.836+0000 7f630ed80640 1 Processor -- start 2024-03-29T11:43:25.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.836+0000 7f630ed80640 1 -- start start 2024-03-29T11:43:25.840 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.836+0000 7f630ed80640 10 monclient: init 2024-03-29T11:43:25.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.837+0000 7f630ed80640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:25.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.837+0000 7f630ed80640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:25.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.837+0000 7f630ed80640 10 monclient(hunting): picked mon.noname-c con 0x7f63080743e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:25.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.837+0000 7f630ed80640 10 monclient(hunting): picked mon.noname-a con 0x7f63080747c0 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:25.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.837+0000 7f630ed80640 10 monclient(hunting): picked mon.noname-b con 0x7f63081436c0 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:25.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.837+0000 7f630ed80640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6308143c30 con 0x7f63080747c0 2024-03-29T11:43:25.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.837+0000 7f630ed80640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6308072560 con 0x7f63080743e0 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.837+0000 7f630ed80640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63080727d0 con 0x7f63081436c0 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.837+0000 7f630ed80640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.837+0000 7f630ed80640 10 monclient(hunting): authenticate will time out at 1326.032418s 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.838+0000 7f630e57f640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f63081436c0 0x7f630814b5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:47598/0 (socket says 172.21.15.100:47598) 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.838+0000 7f630e57f640 1 -- 172.21.15.100:0/416665424 learned_addr learned my addr 172.21.15.100:0/416665424 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.839+0000 7f630cd7c640 1 -- 172.21.15.100:0/416665424 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881226957 0 0) 0x7f63080727d0 con 0x7f63081436c0 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.839+0000 7f630cd7c640 10 monclient(hunting): _negotiate 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.839+0000 7f630cd7c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.839+0000 7f630cd7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.839+0000 7f630cd7c640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.839+0000 7f630cd7c640 1 -- 172.21.15.100:0/416665424 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f4007930 con 0x7f63081436c0 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.839+0000 7f630cd7c640 1 -- 172.21.15.100:0/416665424 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1206760565 0 0) 0x7f62f4007930 con 0x7f63081436c0 2024-03-29T11:43:25.843 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.839+0000 7f630cd7c640 1 -- 172.21.15.100:0/416665424 >> v1:172.21.15.191:6789/0 conn(0x7f63080743e0 legacy=0x7f630813fc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:25.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 1 -- 172.21.15.100:0/416665424 >> v1:172.21.15.100:6789/0 conn(0x7f63080747c0 legacy=0x7f63080ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:25.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: _finish_hunting 0 2024-03-29T11:43:25.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 1 monclient: found mon.noname-b 2024-03-29T11:43:25.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:25.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:43:25.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 1 -- 172.21.15.100:0/416665424 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63081406a0 con 0x7f63081436c0 2024-03-29T11:43:25.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: _finish_auth 0 2024-03-29T11:43:25.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: _check_auth_tickets 2024-03-29T11:43:25.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:25.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 1 -- 172.21.15.100:0/416665424 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6300001a10 con 0x7f63081436c0 2024-03-29T11:43:25.848 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:25.848 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:25.848 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: dump: 2024-03-29T11:43:25.848 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:25.848 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:25.848 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:25.848 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:25.848 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:25.848 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:25.848 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630ed80640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630ed80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 1 -- 172.21.15.100:0/416665424 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6300001b70 con 0x7f63081436c0 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 1 -- 172.21.15.100:0/416665424 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6300001fc0 con 0x7f63081436c0 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.840+0000 7f630cd7c640 10 monclient: dump: 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:25.849 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.841+0000 7f630ed80640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.841+0000 7f630ed80640 4 set_mon_vals no callback set 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.841+0000 7f630ed80640 10 monclient: shutdown 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.841+0000 7f630ed80640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.841+0000 7f630ed80640 1 -- 172.21.15.100:0/416665424 >> v1:172.21.15.100:6790/0 conn(0x7f63081436c0 legacy=0x7f630814b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:25.850 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.843+0000 7f630ed80640 1 -- 172.21.15.100:0/416665424 shutdown_connections 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.843+0000 7f630ed80640 1 -- 172.21.15.100:0/416665424 >> 172.21.15.100:0/416665424 conn(0x7f63080b40b0 msgr2=0x7f63080b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.843+0000 7f630ed80640 1 -- 172.21.15.100:0/416665424 shutdown_connections 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.843+0000 7f630ed80640 1 -- 172.21.15.100:0/416665424 wait complete. 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.844+0000 7f630ed80640 10 monclient: build_initial_monmap 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.844+0000 7f630ed80640 10 monclient: monmap: 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:43:25.851 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:25.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.844+0000 7f630ed80640 1 librados: starting msgr at 2024-03-29T11:43:25.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.844+0000 7f630ed80640 1 librados: starting objecter 2024-03-29T11:43:25.852 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.844+0000 7f630ed80640 1 Processor -- start 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.845+0000 7f630ed80640 1 -- start start 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.845+0000 7f630ed80640 1 librados: setting wanted keys 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.845+0000 7f630ed80640 1 librados: calling monclient init 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.845+0000 7f630ed80640 10 monclient: init 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.846+0000 7f630ed80640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.846+0000 7f630ed80640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.847+0000 7f630ed80640 10 monclient(hunting): picked mon.noname-b con 0x7f63080743e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.847+0000 7f630ed80640 10 monclient(hunting): picked mon.noname-c con 0x7f63081436c0 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.847+0000 7f630ed80640 10 monclient(hunting): picked mon.noname-a con 0x7f63080ac010 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.847+0000 7f630ed80640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63081deab0 con 0x7f63080ac010 2024-03-29T11:43:25.853 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.847+0000 7f630ed80640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63081df650 con 0x7f63080743e0 2024-03-29T11:43:25.854 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.847+0000 7f630ed80640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6308139b20 con 0x7f63081436c0 2024-03-29T11:43:25.854 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.847+0000 7f630ed80640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:25.854 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.847+0000 7f630ed80640 10 monclient(hunting): authenticate will time out at 1326.042931s 2024-03-29T11:43:25.857 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f630d57d640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f63081436c0 0x7f63081def40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:49936/0 (socket says 172.21.15.100:49936) 2024-03-29T11:43:25.857 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f630d57d640 1 -- 172.21.15.100:0/3468122310 learned_addr learned my addr 172.21.15.100:0/3468122310 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:25.857 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 10 client.?.objecter ms_handle_connect 0x7f63081436c0 2024-03-29T11:43:25.857 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:25.857 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137955582 0 0) 0x7f6308139b20 con 0x7f63081436c0 2024-03-29T11:43:25.857 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:25.857 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ec004eb0 con 0x7f63081436c0 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 375061106 0 0) 0x7f62ec004eb0 con 0x7f63081436c0 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6308139b20 con 0x7f63081436c0 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f62f8001960 con 0x7f63081436c0 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.852+0000 7f62fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.853+0000 7f62fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.853+0000 7f62fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:25.858 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:25.859 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:25.859 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:25.859 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:25.859 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:25.859 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:25.859 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:25.859 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:25.859 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.853+0000 7f62fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:25.860 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.853+0000 7f62fe7fc640 10 client.?.objecter ms_handle_connect 0x7f63080ac010 2024-03-29T11:43:25.860 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.853+0000 7f62fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:25.860 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.853+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1856072868 0 0) 0x7f6308139b20 con 0x7f63081436c0 2024-03-29T11:43:25.860 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.853+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 >> v1:172.21.15.191:6789/0 conn(0x7f63080743e0 legacy=0x7f63080aa0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:25.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 >> v1:172.21.15.100:6789/0 conn(0x7f63080ac010 legacy=0x7f6308139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:25.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f62fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:25.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f62fe7fc640 1 monclient: found mon.c 2024-03-29T11:43:25.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f62fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:25.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f62fe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:25.863 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080ab910 con 0x7f63081436c0 2024-03-29T11:43:25.864 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f62fe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:25.864 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f62fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:25.864 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f62fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:25.864 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f630ed80640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:43:25.864 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f630ed80640 10 monclient: _renew_subs 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f630ed80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f630ed80640 1 -- 172.21.15.100:0/3468122310 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63080abbf0 con 0x7f63081436c0 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f630ed80640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f630ed80640 10 monclient: _renew_subs 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f630ed80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f630ed80640 1 -- 172.21.15.100:0/3468122310 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63081de740 con 0x7f63081436c0 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.855+0000 7f630ed80640 1 librados: init done 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.857+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62f8001c80 con 0x7f63081436c0 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.857+0000 7f62fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.857+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f62f8001960 con 0x7f63081436c0 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.857+0000 7f62fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:25.865 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.857+0000 7f62fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.857+0000 7f62fe7fc640 10 monclient: dump: 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:25.866 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.858+0000 7f62fffff640 4 set_mon_vals no callback set 2024-03-29T11:43:25.867 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.858+0000 7f630ed80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:25.867 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.859+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f62f80038a0 con 0x7f63081436c0 2024-03-29T11:43:25.867 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.860+0000 7f630ed80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:25.867 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.860+0000 7f630ed80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:25.868 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.860+0000 7f630ed80640 1 -- 172.21.15.100:0/3468122310 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630804f730 con 0x7f63081436c0 2024-03-29T11:43:25.868 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.860+0000 7f630dd7e640 10 monclient: get_auth_request con 0x7f62ec06ddb0 auth_method 0 2024-03-29T11:43:25.868 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.861+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f62f8084850 con 0x7f63081436c0 2024-03-29T11:43:25.868 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.861+0000 7f62fe7fc640 10 client.4526.objecter ms_dispatch 0x7f6308000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:25.868 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.861+0000 7f62fe7fc640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:25.868 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.861+0000 7f62fe7fc640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:25.868 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.861+0000 7f62fe7fc640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T11:43:25.868 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.861+0000 7f62fe7fc640 10 client.4526.objecter ms_handle_connect 0x7f62ec06ddb0 2024-03-29T11:43:25.872 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.868+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62f804ff60 con 0x7f63081436c0 2024-03-29T11:43:25.872 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.868+0000 7f62fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:25.872 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:25.868+0000 7f62fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:26.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.192+0000 7f630ed80640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:26.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.192+0000 7f630ed80640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:26.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.192+0000 7f630ed80640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:26.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.192+0000 7f630ed80640 1 -- 172.21.15.100:0/3468122310 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6308141510 con 0x7f63081436c0 2024-03-29T11:43:26.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.194+0000 7f62fe7fc640 1 -- 172.21.15.100:0/3468122310 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f62f8052060 con 0x7f63081436c0 2024-03-29T11:43:26.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.194+0000 7f62fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:26.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.194+0000 7f62fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:26.199 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:43:26.200 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:26.203 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.199+0000 7f630ed80640 10 librados: watch_flush enter 2024-03-29T11:43:26.203 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.199+0000 7f630ed80640 10 librados: watch_flush exit 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.199+0000 7f630ed80640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.199+0000 7f630ed80640 10 client.4526.objecter successfully canceled tick 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.199+0000 7f630ed80640 1 -- 172.21.15.100:0/3468122310 >> v1:172.21.15.100:6816/3220289657 conn(0x7f62ec06ddb0 legacy=0x7f62ec070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.199+0000 7f630ed80640 10 monclient: shutdown 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.199+0000 7f630ed80640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.199+0000 7f630ed80640 1 -- 172.21.15.100:0/3468122310 >> v1:172.21.15.100:6790/0 conn(0x7f63081436c0 legacy=0x7f63081def40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.199+0000 7f630ed80640 1 -- 172.21.15.100:0/3468122310 shutdown_connections 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.199+0000 7f630ed80640 1 -- 172.21.15.100:0/3468122310 >> 172.21.15.100:0/3468122310 conn(0x7f63080b40b0 msgr2=0x7f63080b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.200+0000 7f630ed80640 1 -- 172.21.15.100:0/3468122310 shutdown_connections 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.200+0000 7f630ed80640 1 -- 172.21.15.100:0/3468122310 wait complete. 2024-03-29T11:43:26.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:26.200+0000 7f630ed80640 1 librados: shutdown 2024-03-29T11:43:26.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:33.027 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:33.178 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.171+0000 7fa220123640 10 monclient: get_monmap_and_config 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.171+0000 7fa220123640 10 monclient: build_initial_monmap 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.171+0000 7fa220123640 10 monclient: monmap: 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:43:33.172607+0000 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:43:33.172607+0000 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:43:33.179 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.172+0000 7fa220123640 1 Processor -- start 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.172+0000 7fa220123640 1 -- start start 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.172+0000 7fa220123640 10 monclient: init 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.173+0000 7fa220123640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.173+0000 7fa220123640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.173+0000 7fa220123640 10 monclient(hunting): picked mon.noname-c con 0x7fa21806d260 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.173+0000 7fa220123640 10 monclient(hunting): picked mon.noname-a con 0x7fa2180723b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.173+0000 7fa220123640 10 monclient(hunting): picked mon.noname-b con 0x7fa218146b80 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.173+0000 7fa220123640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180701d0 con 0x7fa2180723b0 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.173+0000 7fa220123640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa218070440 con 0x7fa21806d260 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.173+0000 7fa220123640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21806c2c0 con 0x7fa218146b80 2024-03-29T11:43:33.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.173+0000 7fa220123640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:33.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.173+0000 7fa220123640 10 monclient(hunting): authenticate will time out at 1333.368625s 2024-03-29T11:43:33.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.177+0000 7fa21e6c0640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fa218146b80 0x7fa218151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:50966/0 (socket says 172.21.15.100:50966) 2024-03-29T11:43:33.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.177+0000 7fa21e6c0640 1 -- 172.21.15.100:0/124853627 learned_addr learned my addr 172.21.15.100:0/124853627 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:33.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709373993 0 0) 0x7fa2180701d0 con 0x7fa2180723b0 2024-03-29T11:43:33.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa204003520 con 0x7fa2180723b0 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882665242 0 0) 0x7fa218070440 con 0x7fa21806d260 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2180701d0 con 0x7fa21806d260 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055306247 0 0) 0x7fa21806c2c0 con 0x7fa218146b80 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:33.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa218070440 con 0x7fa218146b80 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1411424886 0 0) 0x7fa218070440 con 0x7fa218146b80 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 >> v1:172.21.15.191:6789/0 conn(0x7fa21806d260 legacy=0x7fa218071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 >> v1:172.21.15.100:6789/0 conn(0x7fa2180723b0 legacy=0x7fa218146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 monclient: found mon.noname-b 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21806c7e0 con 0x7fa218146b80 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient: _finish_auth 0 2024-03-29T11:43:33.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa2100019b0 con 0x7fa218146b80 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa21cebd640 10 monclient: dump: 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:33.189 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa220123640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.178+0000 7fa220123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.181+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa210001b10 con 0x7fa218146b80 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.181+0000 7fa21cebd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.181+0000 7fa21cebd640 1 -- 172.21.15.100:0/124853627 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa210001ee0 con 0x7fa218146b80 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.181+0000 7fa21cebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.181+0000 7fa21cebd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.181+0000 7fa21cebd640 10 monclient: dump: 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:33.190 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.181+0000 7fa220123640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.181+0000 7fa220123640 4 set_mon_vals no callback set 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.182+0000 7fa220123640 10 monclient: shutdown 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.182+0000 7fa220123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.182+0000 7fa220123640 1 -- 172.21.15.100:0/124853627 >> v1:172.21.15.100:6790/0 conn(0x7fa218146b80 legacy=0x7fa218151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.183+0000 7fa220123640 1 -- 172.21.15.100:0/124853627 shutdown_connections 2024-03-29T11:43:33.191 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.183+0000 7fa220123640 1 -- 172.21.15.100:0/124853627 >> 172.21.15.100:0/124853627 conn(0x7fa218067500 msgr2=0x7fa2180b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:33.192 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.186+0000 7fa220123640 1 -- 172.21.15.100:0/124853627 shutdown_connections 2024-03-29T11:43:33.192 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.186+0000 7fa220123640 1 -- 172.21.15.100:0/124853627 wait complete. 2024-03-29T11:43:33.193 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.187+0000 7fa220123640 10 monclient: build_initial_monmap 2024-03-29T11:43:33.193 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.187+0000 7fa220123640 10 monclient: monmap: 2024-03-29T11:43:33.193 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:33.193 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:33.193 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:43:33.193 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:43:33.193 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:33.194 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:33.194 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:33.194 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:43:33.194 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:43:33.194 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.187+0000 7fa220123640 1 librados: starting msgr at 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.187+0000 7fa220123640 1 librados: starting objecter 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.187+0000 7fa220123640 1 Processor -- start 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.187+0000 7fa220123640 1 -- start start 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.187+0000 7fa220123640 1 librados: setting wanted keys 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.187+0000 7fa220123640 1 librados: calling monclient init 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.187+0000 7fa220123640 10 monclient: init 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.188+0000 7fa220123640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.188+0000 7fa220123640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.188+0000 7fa220123640 10 monclient(hunting): picked mon.noname-a con 0x7fa21806d260 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.188+0000 7fa220123640 10 monclient(hunting): picked mon.noname-c con 0x7fa2180723b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:33.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.188+0000 7fa220123640 10 monclient(hunting): picked mon.noname-b con 0x7fa218146b80 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:33.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.188+0000 7fa220123640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180aeb10 con 0x7fa21806d260 2024-03-29T11:43:33.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.188+0000 7fa220123640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180aed60 con 0x7fa218146b80 2024-03-29T11:43:33.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.188+0000 7fa220123640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180ad6f0 con 0x7fa2180723b0 2024-03-29T11:43:33.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.188+0000 7fa220123640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:33.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.188+0000 7fa220123640 10 monclient(hunting): authenticate will time out at 1333.383831s 2024-03-29T11:43:33.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.191+0000 7fa21d6be640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fa2180723b0 0x7fa218124a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:53540/0 (socket says 172.21.15.100:53540) 2024-03-29T11:43:33.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.191+0000 7fa21d6be640 1 -- 172.21.15.100:0/1122965519 learned_addr learned my addr 172.21.15.100:0/1122965519 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:33.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.192+0000 7fa20e7fc640 10 client.?.objecter ms_handle_connect 0x7fa2180723b0 2024-03-29T11:43:33.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.192+0000 7fa20e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:33.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.192+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986180682 0 0) 0x7fa2180ad6f0 con 0x7fa2180723b0 2024-03-29T11:43:33.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.194+0000 7fa20e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:33.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.194+0000 7fa20e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:33.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.194+0000 7fa20e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:33.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.194+0000 7fa20e7fc640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:43:33.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.194+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1fc004eb0 con 0x7fa2180723b0 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.194+0000 7fa20e7fc640 10 client.?.objecter ms_handle_connect 0x7fa218146b80 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.194+0000 7fa20e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.195+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3403509999 0 0) 0x7fa1fc004eb0 con 0x7fa2180723b0 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.195+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2180ad6f0 con 0x7fa2180723b0 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.195+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa2140019a0 con 0x7fa2180723b0 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.195+0000 7fa20e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.195+0000 7fa20e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.195+0000 7fa20e7fc640 10 monclient(hunting): dump: 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:33.201 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.195+0000 7fa20e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.195+0000 7fa20e7fc640 10 client.?.objecter ms_handle_connect 0x7fa21806d260 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.195+0000 7fa20e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.196+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1035315198 0 0) 0x7fa2180ad6f0 con 0x7fa2180723b0 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.196+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 >> v1:172.21.15.191:6789/0 conn(0x7fa218146b80 legacy=0x7fa2181ef390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.196+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 >> v1:172.21.15.100:6789/0 conn(0x7fa21806d260 legacy=0x7fa2181e3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.196+0000 7fa20e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.196+0000 7fa20e7fc640 1 monclient: found mon.c 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.196+0000 7fa20e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:33.202 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.196+0000 7fa20e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:33.203 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa218125b90 con 0x7fa2180723b0 2024-03-29T11:43:33.203 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa20e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:33.203 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa20e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:33.203 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa20e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:33.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa220123640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:43:33.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa220123640 10 monclient: _renew_subs 2024-03-29T11:43:33.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa220123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:33.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa220123640 1 -- 172.21.15.100:0/1122965519 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2180ad380 con 0x7fa2180723b0 2024-03-29T11:43:33.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa220123640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:33.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa220123640 10 monclient: _renew_subs 2024-03-29T11:43:33.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa220123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:33.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa220123640 1 -- 172.21.15.100:0/1122965519 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2180af460 con 0x7fa2180723b0 2024-03-29T11:43:33.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa220123640 1 librados: init done 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.197+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa214001ca0 con 0x7fa2180723b0 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.198+0000 7fa20e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.198+0000 7fa20f7fe640 4 set_mon_vals no callback set 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.199+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa214003a40 con 0x7fa2180723b0 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.199+0000 7fa20e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.199+0000 7fa20e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.199+0000 7fa20e7fc640 10 monclient: dump: 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:33.206 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:33.207 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:33.207 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:33.207 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:33.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.199+0000 7fa220123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:33.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.199+0000 7fa220123640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:33.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.199+0000 7fa220123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:33.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.199+0000 7fa220123640 1 -- 172.21.15.100:0/1122965519 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa200005080 con 0x7fa2180723b0 2024-03-29T11:43:33.208 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.205+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fa214003ba0 con 0x7fa2180723b0 2024-03-29T11:43:33.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.206+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fa214084b20 con 0x7fa2180723b0 2024-03-29T11:43:33.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.206+0000 7fa21debf640 10 monclient: get_auth_request con 0x7fa1fc06dbf0 auth_method 0 2024-03-29T11:43:33.210 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.206+0000 7fa20e7fc640 10 client.4532.objecter ms_dispatch 0x7fa218000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:33.210 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.206+0000 7fa20e7fc640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:33.210 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.206+0000 7fa20e7fc640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:33.210 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.206+0000 7fa20e7fc640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T11:43:33.210 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.207+0000 7fa20e7fc640 10 client.4532.objecter ms_handle_connect 0x7fa1fc06dbf0 2024-03-29T11:43:33.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.209+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa214050230 con 0x7fa2180723b0 2024-03-29T11:43:33.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.209+0000 7fa20e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:33.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.209+0000 7fa20e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:33.539 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.535+0000 7fa220123640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:33.539 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.535+0000 7fa220123640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:33.539 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.535+0000 7fa220123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:33.540 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.535+0000 7fa220123640 1 -- 172.21.15.100:0/1122965519 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa200005330 con 0x7fa2180723b0 2024-03-29T11:43:33.540 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:43:33.541 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:33.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.536+0000 7fa20e7fc640 1 -- 172.21.15.100:0/1122965519 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fa214052330 con 0x7fa2180723b0 2024-03-29T11:43:33.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.536+0000 7fa20e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:33.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.536+0000 7fa20e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:33.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.539+0000 7fa220123640 10 librados: watch_flush enter 2024-03-29T11:43:33.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.540+0000 7fa220123640 10 librados: watch_flush exit 2024-03-29T11:43:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.540+0000 7fa220123640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T11:43:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.540+0000 7fa220123640 10 client.4532.objecter successfully canceled tick 2024-03-29T11:43:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.540+0000 7fa220123640 1 -- 172.21.15.100:0/1122965519 >> v1:172.21.15.100:6816/3220289657 conn(0x7fa1fc06dbf0 legacy=0x7fa1fc0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.540+0000 7fa220123640 10 monclient: shutdown 2024-03-29T11:43:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.540+0000 7fa220123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:33.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.540+0000 7fa220123640 1 -- 172.21.15.100:0/1122965519 >> v1:172.21.15.100:6790/0 conn(0x7fa2180723b0 legacy=0x7fa218124a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:33.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.542+0000 7fa220123640 1 -- 172.21.15.100:0/1122965519 shutdown_connections 2024-03-29T11:43:33.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.542+0000 7fa220123640 1 -- 172.21.15.100:0/1122965519 >> 172.21.15.100:0/1122965519 conn(0x7fa218067500 msgr2=0x7fa218149980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:33.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.542+0000 7fa220123640 1 -- 172.21.15.100:0/1122965519 shutdown_connections 2024-03-29T11:43:33.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.542+0000 7fa220123640 1 -- 172.21.15.100:0/1122965519 wait complete. 2024-03-29T11:43:33.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:33.542+0000 7fa220123640 1 librados: shutdown 2024-03-29T11:43:33.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:40.369 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:40.518 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.514+0000 7fbcaa22a640 10 monclient: get_monmap_and_config 2024-03-29T11:43:40.518 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.514+0000 7fbcaa22a640 10 monclient: build_initial_monmap 2024-03-29T11:43:40.519 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.514+0000 7fbcaa22a640 10 monclient: monmap: 2024-03-29T11:43:40.519 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:40.519 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:40.519 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:43:40.515780+0000 2024-03-29T11:43:40.519 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:43:40.515780+0000 2024-03-29T11:43:40.519 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:40.519 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:40.519 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:40.519 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:43:40.519 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:43:40.520 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:40.520 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.515+0000 7fbcaa22a640 1 Processor -- start 2024-03-29T11:43:40.524 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.516+0000 7fbcaa22a640 1 -- start start 2024-03-29T11:43:40.524 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.516+0000 7fbcaa22a640 10 monclient: init 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.516+0000 7fbcaa22a640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.516+0000 7fbcaa22a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.516+0000 7fbcaa22a640 10 monclient(hunting): picked mon.noname-c con 0x7fbca40b0770 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.516+0000 7fbcaa22a640 10 monclient(hunting): picked mon.noname-b con 0x7fbca414a700 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.516+0000 7fbcaa22a640 10 monclient(hunting): picked mon.noname-a con 0x7fbca40ac5a0 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.516+0000 7fbcaa22a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca40aba70 con 0x7fbca40ac5a0 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.516+0000 7fbcaa22a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca40aecb0 con 0x7fbca40b0770 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.516+0000 7fbcaa22a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca40af150 con 0x7fbca414a700 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.517+0000 7fbcaa22a640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.517+0000 7fbcaa22a640 10 monclient(hunting): authenticate will time out at 1340.712088s 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.518+0000 7fbca2ffd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fbca414a700 0x7fbca40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:52820/0 (socket says 172.21.15.100:52820) 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.518+0000 7fbca2ffd640 1 -- 172.21.15.100:0/1320480355 learned_addr learned my addr 172.21.15.100:0/1320480355 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.519+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978759855 0 0) 0x7fbca40af150 con 0x7fbca414a700 2024-03-29T11:43:40.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.519+0000 7fbca27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:40.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.519+0000 7fbca27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:40.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.519+0000 7fbca27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:40.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.519+0000 7fbca27fc640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:43:40.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.519+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc8c003520 con 0x7fbca414a700 2024-03-29T11:43:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283347814 0 0) 0x7fbca40aba70 con 0x7fbca40ac5a0 2024-03-29T11:43:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:43:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca40af150 con 0x7fbca40ac5a0 2024-03-29T11:43:40.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981044298 0 0) 0x7fbca40aecb0 con 0x7fbca40b0770 2024-03-29T11:43:40.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:40.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:40.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:40.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:43:40.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.524+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca40aba70 con 0x7fbca40b0770 2024-03-29T11:43:40.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.526+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 486040496 0 0) 0x7fbca40aba70 con 0x7fbca40b0770 2024-03-29T11:43:40.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.526+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 >> v1:172.21.15.100:6790/0 conn(0x7fbca414a700 legacy=0x7fbca40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:40.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.526+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 >> v1:172.21.15.100:6789/0 conn(0x7fbca40ac5a0 legacy=0x7fbca4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:40.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.526+0000 7fbca27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.526+0000 7fbca27fc640 1 monclient: found mon.noname-c 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.526+0000 7fbca27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.526+0000 7fbca27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.526+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca40ac9d0 con 0x7fbca40b0770 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.527+0000 7fbca27fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.527+0000 7fbca27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.527+0000 7fbca27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.527+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fbc9c001480 con 0x7fbca40b0770 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.527+0000 7fbca27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.527+0000 7fbca27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.527+0000 7fbca27fc640 10 monclient: dump: 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:40.535 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.527+0000 7fbcaa22a640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.527+0000 7fbcaa22a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.528+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc9c0016a0 con 0x7fbca40b0770 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.528+0000 7fbca27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.528+0000 7fbca27fc640 1 -- 172.21.15.100:0/1320480355 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fbc9c001a30 con 0x7fbca40b0770 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.528+0000 7fbca27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:40.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.528+0000 7fbca27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.528+0000 7fbca27fc640 10 monclient: dump: 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.528+0000 7fbcaa22a640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.528+0000 7fbcaa22a640 4 set_mon_vals no callback set 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.529+0000 7fbcaa22a640 10 monclient: shutdown 2024-03-29T11:43:40.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.529+0000 7fbcaa22a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:40.538 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.529+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1320480355 >> v1:172.21.15.191:6789/0 conn(0x7fbca40b0770 legacy=0x7fbca4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:40.540 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.531+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1320480355 shutdown_connections 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.531+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1320480355 >> 172.21.15.100:0/1320480355 conn(0x7fbca406b660 msgr2=0x7fbca406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.531+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1320480355 shutdown_connections 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.531+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1320480355 wait complete. 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.532+0000 7fbcaa22a640 10 monclient: build_initial_monmap 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.532+0000 7fbcaa22a640 10 monclient: monmap: 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:40.541 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.533+0000 7fbcaa22a640 1 librados: starting msgr at 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.533+0000 7fbcaa22a640 1 librados: starting objecter 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.533+0000 7fbcaa22a640 1 Processor -- start 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.533+0000 7fbcaa22a640 1 -- start start 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.533+0000 7fbcaa22a640 1 librados: setting wanted keys 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.533+0000 7fbcaa22a640 1 librados: calling monclient init 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.533+0000 7fbcaa22a640 10 monclient: init 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.534+0000 7fbcaa22a640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.534+0000 7fbcaa22a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.534+0000 7fbcaa22a640 10 monclient(hunting): picked mon.noname-c con 0x7fbca40ac5a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.534+0000 7fbcaa22a640 10 monclient(hunting): picked mon.noname-b con 0x7fbca40b0770 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.534+0000 7fbcaa22a640 10 monclient(hunting): picked mon.noname-a con 0x7fbca414a700 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:40.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.534+0000 7fbcaa22a640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca4128190 con 0x7fbca414a700 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.534+0000 7fbcaa22a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca41283c0 con 0x7fbca40b0770 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.534+0000 7fbcaa22a640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca4126040 con 0x7fbca40ac5a0 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.534+0000 7fbcaa22a640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.534+0000 7fbcaa22a640 10 monclient(hunting): authenticate will time out at 1340.729466s 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.535+0000 7fbca37fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fbca40ac5a0 0x7fbca41450d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:55036/0 (socket says 172.21.15.100:55036) 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.535+0000 7fbca37fe640 1 -- 172.21.15.100:0/1150908724 learned_addr learned my addr 172.21.15.100:0/1150908724 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.535+0000 7fbc8bfff640 10 client.?.objecter ms_handle_connect 0x7fbca40ac5a0 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.535+0000 7fbc8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.535+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652049149 0 0) 0x7fbca4126040 con 0x7fbca40ac5a0 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.535+0000 7fbc8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.535+0000 7fbc8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.535+0000 7fbc8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.535+0000 7fbc8bfff640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:43:40.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.535+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc90004eb0 con 0x7fbca40ac5a0 2024-03-29T11:43:40.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.536+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 16914945 0 0) 0x7fbc90004eb0 con 0x7fbca40ac5a0 2024-03-29T11:43:40.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.536+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbca4126040 con 0x7fbca40ac5a0 2024-03-29T11:43:40.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.539+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fbc9c001480 con 0x7fbca40ac5a0 2024-03-29T11:43:40.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.539+0000 7fbc8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:40.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:40.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 10 monclient(hunting): dump: 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 10 client.?.objecter ms_handle_connect 0x7fbca414a700 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:40.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017137759 0 0) 0x7fbca4128190 con 0x7fbca414a700 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.540+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc90004eb0 con 0x7fbca414a700 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.541+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3486732701 0 0) 0x7fbca4126040 con 0x7fbca40ac5a0 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.541+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 >> v1:172.21.15.191:6789/0 conn(0x7fbca40b0770 legacy=0x7fbca41dbad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.541+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 >> v1:172.21.15.100:6789/0 conn(0x7fbca414a700 legacy=0x7fbca41e7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.541+0000 7fbc8bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.541+0000 7fbc8bfff640 1 monclient: found mon.c 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.541+0000 7fbc8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.541+0000 7fbc8bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.541+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca4126f70 con 0x7fbca40ac5a0 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.541+0000 7fbc8bfff640 10 monclient: _finish_auth 0 2024-03-29T11:43:40.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbc8bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbc8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbcaa22a640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:43:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbcaa22a640 10 monclient: _renew_subs 2024-03-29T11:43:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbcaa22a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1150908724 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca4127430 con 0x7fbca40ac5a0 2024-03-29T11:43:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbcaa22a640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbcaa22a640 10 monclient: _renew_subs 2024-03-29T11:43:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbcaa22a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1150908724 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca41459c0 con 0x7fbca40ac5a0 2024-03-29T11:43:40.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.542+0000 7fbcaa22a640 1 librados: init done 2024-03-29T11:43:40.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc9c0038a0 con 0x7fbca40ac5a0 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbc8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbcaa22a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fbc9c003cb0 con 0x7fbca40ac5a0 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbcaa22a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbcaa22a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1150908724 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca40ab560 con 0x7fbca40ac5a0 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbc8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbc8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbc8bfff640 10 monclient: dump: 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:40.551 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:40.552 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:40.552 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:40.552 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:40.552 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:40.552 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:40.553 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.546+0000 7fbca1ffb640 4 set_mon_vals no callback set 2024-03-29T11:43:40.554 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.550+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fbc9c003e10 con 0x7fbca40ac5a0 2024-03-29T11:43:40.554 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.551+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fbc9c084bc0 con 0x7fbca40ac5a0 2024-03-29T11:43:40.555 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.551+0000 7fbc8bfff640 10 client.4538.objecter ms_dispatch 0x7fbca4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:40.555 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.552+0000 7fbc8bfff640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:40.555 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.552+0000 7fbc8bfff640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:40.555 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.552+0000 7fbc8bfff640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T11:43:40.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.552+0000 7fbca2ffd640 10 monclient: get_auth_request con 0x7fbc9006daf0 auth_method 0 2024-03-29T11:43:40.556 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.553+0000 7fbc8bfff640 10 client.4538.objecter ms_handle_connect 0x7fbc9006daf0 2024-03-29T11:43:40.563 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.560+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc9c0502d0 con 0x7fbca40ac5a0 2024-03-29T11:43:40.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.560+0000 7fbc8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:40.564 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.560+0000 7fbc8bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:40.883 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.879+0000 7fbcaa22a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:40.883 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.879+0000 7fbcaa22a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:40.883 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.879+0000 7fbcaa22a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:40.883 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.879+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1150908724 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbca41db710 con 0x7fbca40ac5a0 2024-03-29T11:43:40.883 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.880+0000 7fbc8bfff640 1 -- 172.21.15.100:0/1150908724 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fbc9c0523d0 con 0x7fbca40ac5a0 2024-03-29T11:43:40.884 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.880+0000 7fbc8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:40.884 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.880+0000 7fbc8bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:40.884 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:43:40.884 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:40.887 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.883+0000 7fbcaa22a640 10 librados: watch_flush enter 2024-03-29T11:43:40.887 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.884+0000 7fbcaa22a640 10 librados: watch_flush exit 2024-03-29T11:43:40.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.884+0000 7fbcaa22a640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T11:43:40.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.884+0000 7fbcaa22a640 10 client.4538.objecter successfully canceled tick 2024-03-29T11:43:40.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.885+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1150908724 >> v1:172.21.15.100:6816/3220289657 conn(0x7fbc9006daf0 legacy=0x7fbc9006ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:40.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.885+0000 7fbcaa22a640 10 monclient: shutdown 2024-03-29T11:43:40.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.885+0000 7fbcaa22a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:40.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.885+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1150908724 >> v1:172.21.15.100:6790/0 conn(0x7fbca40ac5a0 legacy=0x7fbca41450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:40.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.887+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1150908724 shutdown_connections 2024-03-29T11:43:40.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.887+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1150908724 >> 172.21.15.100:0/1150908724 conn(0x7fbca406b660 msgr2=0x7fbca40abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:40.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.890+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1150908724 shutdown_connections 2024-03-29T11:43:40.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.890+0000 7fbcaa22a640 1 -- 172.21.15.100:0/1150908724 wait complete. 2024-03-29T11:43:40.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:40.890+0000 7fbcaa22a640 1 librados: shutdown 2024-03-29T11:43:40.910 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:47.717 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:47.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.884+0000 7f4d17fff640 10 monclient: get_monmap_and_config 2024-03-29T11:43:47.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.884+0000 7f4d17fff640 10 monclient: build_initial_monmap 2024-03-29T11:43:47.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.884+0000 7f4d17fff640 10 monclient: monmap: 2024-03-29T11:43:47.889 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:47.889 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:47.889 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:43:47.885173+0000 2024-03-29T11:43:47.889 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:43:47.885173+0000 2024-03-29T11:43:47.889 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:47.889 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:47.889 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:47.889 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:43:47.889 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:43:47.889 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:47.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.886+0000 7f4d17fff640 1 Processor -- start 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.886+0000 7f4d17fff640 1 -- start start 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.886+0000 7f4d17fff640 10 monclient: init 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.886+0000 7f4d17fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.887+0000 7f4d17fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.887+0000 7f4d17fff640 10 monclient(hunting): picked mon.noname-a con 0x7f4d10006890 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.887+0000 7f4d17fff640 10 monclient(hunting): picked mon.noname-b con 0x7f4d10006c70 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.887+0000 7f4d17fff640 10 monclient(hunting): picked mon.noname-c con 0x7f4d1002b4c0 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.887+0000 7f4d17fff640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1002c830 con 0x7f4d10006890 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.887+0000 7f4d17fff640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1002afa0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.887+0000 7f4d17fff640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1002aba0 con 0x7f4d10006c70 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.887+0000 7f4d17fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:47.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.887+0000 7f4d17fff640 10 monclient(hunting): authenticate will time out at 1348.082168s 2024-03-29T11:43:47.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.889+0000 7f4d167fc640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f4d10006c70 0x7f4d100c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:59818/0 (socket says 172.21.15.100:59818) 2024-03-29T11:43:47.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.889+0000 7f4d167fc640 1 -- 172.21.15.100:0/4067602120 learned_addr learned my addr 172.21.15.100:0/4067602120 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:47.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.890+0000 7f4d15ffb640 1 -- 172.21.15.100:0/4067602120 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380354353 0 0) 0x7f4d1002aba0 con 0x7f4d10006c70 2024-03-29T11:43:47.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.890+0000 7f4d15ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:47.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.890+0000 7f4d15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:47.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.890+0000 7f4d15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:47.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.890+0000 7f4d15ffb640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:43:47.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.890+0000 7f4d15ffb640 1 -- 172.21.15.100:0/4067602120 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d04003520 con 0x7f4d10006c70 2024-03-29T11:43:47.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.890+0000 7f4d15ffb640 1 -- 172.21.15.100:0/4067602120 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2660073719 0 0) 0x7f4d04003520 con 0x7f4d10006c70 2024-03-29T11:43:47.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.890+0000 7f4d15ffb640 1 -- 172.21.15.100:0/4067602120 >> v1:172.21.15.191:6789/0 conn(0x7f4d1002b4c0 legacy=0x7f4d100cf0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:47.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 1 -- 172.21.15.100:0/4067602120 >> v1:172.21.15.100:6789/0 conn(0x7f4d10006890 legacy=0x7f4d100c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:47.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 1 monclient: found mon.noname-b 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 1 -- 172.21.15.100:0/4067602120 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1002ad30 con 0x7f4d10006c70 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 1 -- 172.21.15.100:0/4067602120 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4d00001670 con 0x7f4d10006c70 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 10 monclient: dump: 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:47.900 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d17fff640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d17fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 1 -- 172.21.15.100:0/4067602120 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d000018d0 con 0x7f4d10006c70 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d17fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.892+0000 7f4d17fff640 4 set_mon_vals no callback set 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.893+0000 7f4d17fff640 10 monclient: shutdown 2024-03-29T11:43:47.901 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.893+0000 7f4d17fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.893+0000 7f4d17fff640 1 -- 172.21.15.100:0/4067602120 >> v1:172.21.15.100:6790/0 conn(0x7f4d10006c70 legacy=0x7f4d100c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.893+0000 7f4d15ffb640 1 -- 172.21.15.100:0/4067602120 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4d00001ca0 con 0x7f4d10006c70 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.893+0000 7f4d15ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.893+0000 7f4d17fff640 1 -- 172.21.15.100:0/4067602120 shutdown_connections 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.893+0000 7f4d17fff640 1 -- 172.21.15.100:0/4067602120 >> 172.21.15.100:0/4067602120 conn(0x7f4d100058f0 msgr2=0x7f4d10005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.893+0000 7f4d17fff640 1 -- 172.21.15.100:0/4067602120 shutdown_connections 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.894+0000 7f4d17fff640 1 -- 172.21.15.100:0/4067602120 wait complete. 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.895+0000 7f4d17fff640 10 monclient: build_initial_monmap 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.895+0000 7f4d17fff640 10 monclient: monmap: 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:43:47.902 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:43:47.903 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:47.903 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:47.903 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:47.903 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:43:47.903 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:43:47.903 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:47.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.896+0000 7f4d17fff640 1 librados: starting msgr at 2024-03-29T11:43:47.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.896+0000 7f4d17fff640 1 librados: starting objecter 2024-03-29T11:43:47.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.896+0000 7f4d17fff640 1 Processor -- start 2024-03-29T11:43:47.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.896+0000 7f4d17fff640 1 -- start start 2024-03-29T11:43:47.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.896+0000 7f4d17fff640 1 librados: setting wanted keys 2024-03-29T11:43:47.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.896+0000 7f4d17fff640 1 librados: calling monclient init 2024-03-29T11:43:47.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.896+0000 7f4d17fff640 10 monclient: init 2024-03-29T11:43:47.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.898+0000 7f4d17fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:47.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.898+0000 7f4d17fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:47.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.898+0000 7f4d17fff640 10 monclient(hunting): picked mon.noname-b con 0x7f4d10006890 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:47.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.898+0000 7f4d17fff640 10 monclient(hunting): picked mon.noname-c con 0x7f4d1002b4c0 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:47.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.898+0000 7f4d17fff640 10 monclient(hunting): picked mon.noname-a con 0x7f4d100c1a90 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:47.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.898+0000 7f4d17fff640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d100c15c0 con 0x7f4d100c1a90 2024-03-29T11:43:47.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.898+0000 7f4d17fff640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d100c3e10 con 0x7f4d10006890 2024-03-29T11:43:47.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.898+0000 7f4d17fff640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d100bdcf0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.898+0000 7f4d17fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:47.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.898+0000 7f4d17fff640 10 monclient(hunting): authenticate will time out at 1348.093604s 2024-03-29T11:43:47.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.903+0000 7f4d167fc640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f4d1002b4c0 0x7f4d100c24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:60428/0 (socket says 172.21.15.100:60428) 2024-03-29T11:43:47.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.903+0000 7f4d167fc640 1 -- 172.21.15.100:0/1644931946 learned_addr learned my addr 172.21.15.100:0/1644931946 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:47.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 client.?.objecter ms_handle_connect 0x7f4d1002b4c0 2024-03-29T11:43:47.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:47.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656121370 0 0) 0x7f4d100bdcf0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:47.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:47.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:47.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:43:47.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cf8004eb0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 client.?.objecter ms_handle_connect 0x7f4d10006890 2024-03-29T11:43:47.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:47.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202427864 0 0) 0x7f4d100c3e10 con 0x7f4d10006890 2024-03-29T11:43:47.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:47.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:47.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d100bdcf0 con 0x7f4d10006890 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4195951888 0 0) 0x7f4cf8004eb0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d100c3e10 con 0x7f4d1002b4c0 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4d00001be0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): dump: 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:47.912 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.904+0000 7f4d0ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1481305421 0 0) 0x7f4d100c3e10 con 0x7f4d1002b4c0 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 >> v1:172.21.15.191:6789/0 conn(0x7f4d10006890 legacy=0x7f4d100bd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 >> v1:172.21.15.100:6789/0 conn(0x7f4d100c1a90 legacy=0x7f4d100be180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 1 monclient: found mon.c 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d100c2f30 con 0x7f4d1002b4c0 2024-03-29T11:43:47.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 10 monclient: _finish_auth 0 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d17fff640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d17fff640 10 monclient: _renew_subs 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d17fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d17fff640 1 -- 172.21.15.100:0/1644931946 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d100c2bd0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d00004bc0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f4d00004fd0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d17fff640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:47.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d0ffff640 10 monclient: dump: 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d17fff640 10 monclient: _renew_subs 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d17fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d17fff640 1 -- 172.21.15.100:0/1644931946 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d100c4ad0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d17fff640 1 librados: init done 2024-03-29T11:43:47.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.905+0000 7f4d14ff9640 4 set_mon_vals no callback set 2024-03-29T11:43:47.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.907+0000 7f4d17fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:47.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.907+0000 7f4d17fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:47.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.907+0000 7f4d17fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:47.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.907+0000 7f4d17fff640 1 -- 172.21.15.100:0/1644931946 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d10004cd0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.910+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f4d00005130 con 0x7f4d1002b4c0 2024-03-29T11:43:47.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.911+0000 7f4d16ffd640 10 monclient: get_auth_request con 0x7f4cf807eb80 auth_method 0 2024-03-29T11:43:47.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.911+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f4d00084da0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.911+0000 7f4d0ffff640 10 client.4544.objecter ms_dispatch 0x7f4d18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:47.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.911+0000 7f4d0ffff640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:47.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.911+0000 7f4d0ffff640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:47.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.911+0000 7f4d0ffff640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T11:43:47.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.911+0000 7f4d0ffff640 10 client.4544.objecter ms_handle_connect 0x7f4cf807eb80 2024-03-29T11:43:47.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.921+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d000504b0 con 0x7f4d1002b4c0 2024-03-29T11:43:47.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.921+0000 7f4d0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:47.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:47.921+0000 7f4d0ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:48.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.293+0000 7f4d17fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:48.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.293+0000 7f4d17fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:48.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.293+0000 7f4d17fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:48.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.293+0000 7f4d17fff640 1 -- 172.21.15.100:0/1644931946 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d100c2d60 con 0x7f4d1002b4c0 2024-03-29T11:43:48.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.295+0000 7f4d0ffff640 1 -- 172.21.15.100:0/1644931946 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f4d000525b0 con 0x7f4d1002b4c0 2024-03-29T11:43:48.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.295+0000 7f4d0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:48.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.295+0000 7f4d0ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:48.302 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:43:48.303 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:48.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.299+0000 7f4d17fff640 10 librados: watch_flush enter 2024-03-29T11:43:48.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.299+0000 7f4d17fff640 10 librados: watch_flush exit 2024-03-29T11:43:48.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.299+0000 7f4d17fff640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T11:43:48.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.299+0000 7f4d17fff640 10 client.4544.objecter successfully canceled tick 2024-03-29T11:43:48.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.299+0000 7f4d17fff640 1 -- 172.21.15.100:0/1644931946 >> v1:172.21.15.100:6816/3220289657 conn(0x7f4cf807eb80 legacy=0x7f4cf8081040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:48.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.300+0000 7f4d17fff640 10 monclient: shutdown 2024-03-29T11:43:48.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.300+0000 7f4d17fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:48.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.300+0000 7f4d17fff640 1 -- 172.21.15.100:0/1644931946 >> v1:172.21.15.100:6790/0 conn(0x7f4d1002b4c0 legacy=0x7f4d100c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:48.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.300+0000 7f4d17fff640 1 -- 172.21.15.100:0/1644931946 shutdown_connections 2024-03-29T11:43:48.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.300+0000 7f4d17fff640 1 -- 172.21.15.100:0/1644931946 >> 172.21.15.100:0/1644931946 conn(0x7f4d100058f0 msgr2=0x7f4d10038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:48.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.302+0000 7f4d17fff640 1 -- 172.21.15.100:0/1644931946 shutdown_connections 2024-03-29T11:43:48.305 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.302+0000 7f4d17fff640 1 -- 172.21.15.100:0/1644931946 wait complete. 2024-03-29T11:43:48.306 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:48.303+0000 7f4d17fff640 1 librados: shutdown 2024-03-29T11:43:48.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:55.133 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:55.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.281+0000 7f5371ea8640 10 monclient: get_monmap_and_config 2024-03-29T11:43:55.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.281+0000 7f5371ea8640 10 monclient: build_initial_monmap 2024-03-29T11:43:55.285 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.281+0000 7f5371ea8640 10 monclient: monmap: 2024-03-29T11:43:55.285 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:55.285 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:55.285 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:43:55.282116+0000 2024-03-29T11:43:55.285 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:43:55.282116+0000 2024-03-29T11:43:55.286 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:55.286 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:55.286 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:55.286 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:43:55.286 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:43:55.286 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:55.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.285+0000 7f5371ea8640 1 Processor -- start 2024-03-29T11:43:55.288 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.285+0000 7f5371ea8640 1 -- start start 2024-03-29T11:43:55.289 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.285+0000 7f5371ea8640 10 monclient: init 2024-03-29T11:43:55.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f5371ea8640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:55.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f5371ea8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:55.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f5371ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f536c0b4710 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:55.291 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f5371ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f536c0b3970 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f5371ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f536c13f390 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f5371ea8640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c0af020 con 0x7f536c13f390 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f5371ea8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c0b0910 con 0x7f536c0b3970 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f5371ea8640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c0b0bf0 con 0x7f536c0b4710 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f5371ea8640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f5371ea8640 10 monclient(hunting): authenticate will time out at 1355.481462s 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f536b7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f536c0b4710 0x7f536c0b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:54606/0 (socket says 172.21.15.100:54606) 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.286+0000 7f536b7fe640 1 -- 172.21.15.100:0/3466092241 learned_addr learned my addr 172.21.15.100:0/3466092241 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.287+0000 7f536a7fc640 1 -- 172.21.15.100:0/3466092241 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307267923 0 0) 0x7f536c0b0bf0 con 0x7f536c0b4710 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.287+0000 7f536a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.287+0000 7f536a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.287+0000 7f536a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:55.292 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.287+0000 7f536a7fc640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:43:55.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.287+0000 7f536a7fc640 1 -- 172.21.15.100:0/3466092241 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5354003520 con 0x7f536c0b4710 2024-03-29T11:43:55.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.288+0000 7f536a7fc640 1 -- 172.21.15.100:0/3466092241 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2943088899 0 0) 0x7f5354003520 con 0x7f536c0b4710 2024-03-29T11:43:55.293 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.288+0000 7f536a7fc640 1 -- 172.21.15.100:0/3466092241 >> v1:172.21.15.191:6789/0 conn(0x7f536c0b3970 legacy=0x7f536c0af4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:55.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.290+0000 7f536a7fc640 1 -- 172.21.15.100:0/3466092241 >> v1:172.21.15.100:6789/0 conn(0x7f536c13f390 legacy=0x7f536c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:55.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.290+0000 7f536a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:55.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.290+0000 7f536a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:43:55.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.290+0000 7f536a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:55.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f536a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:43:55.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f536a7fc640 1 -- 172.21.15.100:0/3466092241 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c0aeb00 con 0x7f536c0b4710 2024-03-29T11:43:55.295 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f536a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f536a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f536a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f536a7fc640 1 -- 172.21.15.100:0/3466092241 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5364001a10 con 0x7f536c0b4710 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f536a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f536a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f536a7fc640 10 monclient: dump: 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:55.296 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:55.297 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:55.297 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:55.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f5371ea8640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:43:55.297 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.291+0000 7f5371ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:55.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.292+0000 7f536a7fc640 1 -- 172.21.15.100:0/3466092241 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5364001be0 con 0x7f536c0b4710 2024-03-29T11:43:55.298 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.292+0000 7f536a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:55.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.292+0000 7f5371ea8640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:55.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.292+0000 7f5371ea8640 4 set_mon_vals no callback set 2024-03-29T11:43:55.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.293+0000 7f5371ea8640 10 monclient: shutdown 2024-03-29T11:43:55.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.293+0000 7f5371ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:55.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.293+0000 7f5371ea8640 1 -- 172.21.15.100:0/3466092241 >> v1:172.21.15.100:6790/0 conn(0x7f536c0b4710 legacy=0x7f536c0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:55.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.293+0000 7f5371ea8640 1 -- 172.21.15.100:0/3466092241 shutdown_connections 2024-03-29T11:43:55.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.293+0000 7f5371ea8640 1 -- 172.21.15.100:0/3466092241 >> 172.21.15.100:0/3466092241 conn(0x7f536c066a00 msgr2=0x7f536c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:55.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.294+0000 7f5371ea8640 1 -- 172.21.15.100:0/3466092241 shutdown_connections 2024-03-29T11:43:55.299 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.294+0000 7f5371ea8640 1 -- 172.21.15.100:0/3466092241 wait complete. 2024-03-29T11:43:55.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.295+0000 7f5371ea8640 10 monclient: build_initial_monmap 2024-03-29T11:43:55.300 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.295+0000 7f5371ea8640 10 monclient: monmap: 2024-03-29T11:43:55.300 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:43:55.300 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:55.300 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:43:55.300 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:43:55.300 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:55.300 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:55.301 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:43:55.301 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:43:55.301 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:43:55.301 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:55.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.296+0000 7f5371ea8640 1 librados: starting msgr at 2024-03-29T11:43:55.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.296+0000 7f5371ea8640 1 librados: starting objecter 2024-03-29T11:43:55.301 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.296+0000 7f5371ea8640 1 Processor -- start 2024-03-29T11:43:55.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 1 -- start start 2024-03-29T11:43:55.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 1 librados: setting wanted keys 2024-03-29T11:43:55.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 1 librados: calling monclient init 2024-03-29T11:43:55.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 10 monclient: init 2024-03-29T11:43:55.303 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:55.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:55.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f536c0b3970 addr v1:172.21.15.100:6789/0 2024-03-29T11:43:55.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f536c13f390 addr v1:172.21.15.100:6790/0 2024-03-29T11:43:55.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f536c06de90 addr v1:172.21.15.191:6789/0 2024-03-29T11:43:55.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c14e3d0 con 0x7f536c0b3970 2024-03-29T11:43:55.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c1df7a0 con 0x7f536c06de90 2024-03-29T11:43:55.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c06ed80 con 0x7f536c13f390 2024-03-29T11:43:55.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:55.304 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.297+0000 7f5371ea8640 10 monclient(hunting): authenticate will time out at 1355.492965s 2024-03-29T11:43:55.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.301+0000 7f536affd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f536c13f390 0x7f536c1de980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:56824/0 (socket says 172.21.15.100:56824) 2024-03-29T11:43:55.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.301+0000 7f536affd640 1 -- 172.21.15.100:0/444209608 learned_addr learned my addr 172.21.15.100:0/444209608 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:43:55.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 10 client.?.objecter ms_handle_connect 0x7f536c13f390 2024-03-29T11:43:55.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:55.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612629841 0 0) 0x7f536c06ed80 con 0x7f536c13f390 2024-03-29T11:43:55.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:55.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:55.308 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:55.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:43:55.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5358004eb0 con 0x7f536c13f390 2024-03-29T11:43:55.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 10 client.?.objecter ms_handle_connect 0x7f536c06de90 2024-03-29T11:43:55.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:55.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4162017425 0 0) 0x7f5358004eb0 con 0x7f536c13f390 2024-03-29T11:43:55.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.302+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f536c06ed80 con 0x7f536c13f390 2024-03-29T11:43:55.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5360001870 con 0x7f536c13f390 2024-03-29T11:43:55.309 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): dump: 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:55.311 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491352760 0 0) 0x7f536c1df7a0 con 0x7f536c06de90 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5358004eb0 con 0x7f536c06de90 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2314813739 0 0) 0x7f5358004eb0 con 0x7f536c06de90 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f536c1df7a0 con 0x7f536c06de90 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f535c001690 con 0x7f536c06de90 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient(hunting): dump: 2024-03-29T11:43:55.312 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 519091439 0 0) 0x7f536c06ed80 con 0x7f536c13f390 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 >> v1:172.21.15.191:6789/0 conn(0x7f536c06de90 legacy=0x7f536c1df090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 >> v1:172.21.15.100:6789/0 conn(0x7f536c0b3970 legacy=0x7f536c14d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 monclient: found mon.c 2024-03-29T11:43:55.313 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c06f060 con 0x7f536c13f390 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient: _finish_auth 0 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.303+0000 7f5353fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.304+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5360001e00 con 0x7f536c13f390 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.304+0000 7f5353fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.304+0000 7f5369ffb640 4 set_mon_vals no callback set 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5360004aa0 con 0x7f536c13f390 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5353fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5353fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5353fff640 10 monclient: dump: 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:43:55.314 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5371ea8640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5371ea8640 10 monclient: _renew_subs 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5371ea8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5371ea8640 1 -- 172.21.15.100:0/444209608 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c06f340 con 0x7f536c13f390 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5371ea8640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:55.315 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5371ea8640 10 monclient: _renew_subs 2024-03-29T11:43:55.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5371ea8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:55.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5371ea8640 1 -- 172.21.15.100:0/444209608 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c06f940 con 0x7f536c13f390 2024-03-29T11:43:55.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.306+0000 7f5371ea8640 1 librados: init done 2024-03-29T11:43:55.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.310+0000 7f5371ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:55.316 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.310+0000 7f5371ea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:55.317 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.310+0000 7f5371ea8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:55.317 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.310+0000 7f5371ea8640 1 -- 172.21.15.100:0/444209608 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536c0688a0 con 0x7f536c13f390 2024-03-29T11:43:55.318 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.315+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f536001aa70 con 0x7f536c13f390 2024-03-29T11:43:55.320 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.316+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f5360084b10 con 0x7f536c13f390 2024-03-29T11:43:55.321 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.317+0000 7f5353fff640 10 client.4550.objecter ms_dispatch 0x7f536c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:55.321 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.317+0000 7f536b7fe640 10 monclient: get_auth_request con 0x7f535806df10 auth_method 0 2024-03-29T11:43:55.321 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.317+0000 7f5353fff640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:55.321 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.317+0000 7f5353fff640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:55.321 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.317+0000 7f5353fff640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T11:43:55.321 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.317+0000 7f5353fff640 10 client.4550.objecter ms_handle_connect 0x7f535806df10 2024-03-29T11:43:55.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.327+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5360050220 con 0x7f536c13f390 2024-03-29T11:43:55.331 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.327+0000 7f5353fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:55.331 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.327+0000 7f5353fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:55.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.681+0000 7f5371ea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:55.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.681+0000 7f5371ea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:55.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.681+0000 7f5371ea8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:43:55.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.681+0000 7f5371ea8640 1 -- 172.21.15.100:0/444209608 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f536c14e580 con 0x7f536c13f390 2024-03-29T11:43:55.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.682+0000 7f5353fff640 1 -- 172.21.15.100:0/444209608 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f536c14e580 con 0x7f536c13f390 2024-03-29T11:43:55.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.682+0000 7f5353fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:55.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.682+0000 7f5353fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:55.687 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:43:55.687 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:55.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.686+0000 7f5371ea8640 10 librados: watch_flush enter 2024-03-29T11:43:55.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.686+0000 7f5371ea8640 10 librados: watch_flush exit 2024-03-29T11:43:55.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.686+0000 7f5371ea8640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T11:43:55.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.686+0000 7f5371ea8640 10 client.4550.objecter successfully canceled tick 2024-03-29T11:43:55.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.686+0000 7f5371ea8640 1 -- 172.21.15.100:0/444209608 >> v1:172.21.15.100:6816/3220289657 conn(0x7f535806df10 legacy=0x7f53580703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:55.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.686+0000 7f5371ea8640 10 monclient: shutdown 2024-03-29T11:43:55.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.686+0000 7f5371ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:55.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.686+0000 7f5371ea8640 1 -- 172.21.15.100:0/444209608 >> v1:172.21.15.100:6790/0 conn(0x7f536c13f390 legacy=0x7f536c1de980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:55.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.686+0000 7f5371ea8640 1 -- 172.21.15.100:0/444209608 shutdown_connections 2024-03-29T11:43:55.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.686+0000 7f5371ea8640 1 -- 172.21.15.100:0/444209608 >> 172.21.15.100:0/444209608 conn(0x7f536c066a00 msgr2=0x7f536c0b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:55.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.687+0000 7f5371ea8640 1 -- 172.21.15.100:0/444209608 shutdown_connections 2024-03-29T11:43:55.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.687+0000 7f5371ea8640 1 -- 172.21.15.100:0/444209608 wait complete. 2024-03-29T11:43:55.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:43:55.687+0000 7f5371ea8640 1 librados: shutdown 2024-03-29T11:43:55.709 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:02.517 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:02.667 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.663+0000 7f36bb3cf640 10 monclient: get_monmap_and_config 2024-03-29T11:44:02.667 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.663+0000 7f36bb3cf640 10 monclient: build_initial_monmap 2024-03-29T11:44:02.667 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.663+0000 7f36bb3cf640 10 monclient: monmap: 2024-03-29T11:44:02.667 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:02.667 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:02.668 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:44:02.664695+0000 2024-03-29T11:44:02.668 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:44:02.664695+0000 2024-03-29T11:44:02.668 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:02.668 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:02.668 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:02.668 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:44:02.668 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:44:02.668 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:02.669 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.664+0000 7f36bb3cf640 1 Processor -- start 2024-03-29T11:44:02.669 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.665+0000 7f36bb3cf640 1 -- start start 2024-03-29T11:44:02.669 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.665+0000 7f36bb3cf640 10 monclient: init 2024-03-29T11:44:02.670 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36bb3cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:02.670 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36bb3cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:02.670 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36bb3cf640 10 monclient(hunting): picked mon.noname-c con 0x7f36b41457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:02.670 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36bb3cf640 10 monclient(hunting): picked mon.noname-b con 0x7f36b413f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:02.670 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36bb3cf640 10 monclient(hunting): picked mon.noname-a con 0x7f36b40a9d70 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:02.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36bb3cf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40aaef0 con 0x7f36b40a9d70 2024-03-29T11:44:02.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36bb3cf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b4145bd0 con 0x7f36b41457f0 2024-03-29T11:44:02.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36bb3cf640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b4145eb0 con 0x7f36b413f650 2024-03-29T11:44:02.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36bb3cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:02.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36bb3cf640 10 monclient(hunting): authenticate will time out at 1362.861469s 2024-03-29T11:44:02.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36b896a640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f36b413f650 0x7f36b40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:60102/0 (socket says 172.21.15.100:60102) 2024-03-29T11:44:02.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.666+0000 7f36b896a640 1 -- 172.21.15.100:0/2616636820 learned_addr learned my addr 172.21.15.100:0/2616636820 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:02.675 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596015874 0 0) 0x7f36b4145eb0 con 0x7f36b413f650 2024-03-29T11:44:02.675 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a0003520 con 0x7f36b413f650 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153442340 0 0) 0x7f36b40aaef0 con 0x7f36b40a9d70 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.667+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b4145eb0 con 0x7f36b40a9d70 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 51548601 0 0) 0x7f36a0003520 con 0x7f36b413f650 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 >> v1:172.21.15.191:6789/0 conn(0x7f36b41457f0 legacy=0x7f36b413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:02.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 >> v1:172.21.15.100:6789/0 conn(0x7f36b40a9d70 legacy=0x7f36b414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 1 monclient: found mon.noname-b 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4146340 con 0x7f36b413f650 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: _finish_auth 0 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f36a4001690 con 0x7f36b413f650 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: dump: 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:02.677 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36bb3cf640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36bb3cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a40017f0 con 0x7f36b413f650 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 1 -- 172.21.15.100:0/2616636820 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f36a4001c00 con 0x7f36b413f650 2024-03-29T11:44:02.678 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36abfff640 10 monclient: dump: 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36bb3cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:02.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.668+0000 7f36bb3cf640 4 set_mon_vals no callback set 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.669+0000 7f36bb3cf640 10 monclient: shutdown 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.669+0000 7f36bb3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.669+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2616636820 >> v1:172.21.15.100:6790/0 conn(0x7f36b413f650 legacy=0x7f36b40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.669+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2616636820 shutdown_connections 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.669+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2616636820 >> 172.21.15.100:0/2616636820 conn(0x7f36b4066a00 msgr2=0x7f36b4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.669+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2616636820 shutdown_connections 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.670+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2616636820 wait complete. 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.670+0000 7f36bb3cf640 10 monclient: build_initial_monmap 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.670+0000 7f36bb3cf640 10 monclient: monmap: 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:44:02.680 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.671+0000 7f36bb3cf640 1 librados: starting msgr at 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.671+0000 7f36bb3cf640 1 librados: starting objecter 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.671+0000 7f36bb3cf640 1 Processor -- start 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.671+0000 7f36bb3cf640 1 -- start start 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.671+0000 7f36bb3cf640 1 librados: setting wanted keys 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.671+0000 7f36bb3cf640 1 librados: calling monclient init 2024-03-29T11:44:02.681 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.671+0000 7f36bb3cf640 10 monclient: init 2024-03-29T11:44:02.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.672+0000 7f36bb3cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.672+0000 7f36bb3cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.672+0000 7f36bb3cf640 10 monclient(hunting): picked mon.noname-c con 0x7f36b40a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.672+0000 7f36bb3cf640 10 monclient(hunting): picked mon.noname-a con 0x7f36b413f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.672+0000 7f36bb3cf640 10 monclient(hunting): picked mon.noname-b con 0x7f36b41457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.672+0000 7f36bb3cf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b41db500 con 0x7f36b413f650 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.673+0000 7f36bb3cf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b41db7c0 con 0x7f36b41457f0 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.673+0000 7f36bb3cf640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b41cd330 con 0x7f36b40a9d70 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.673+0000 7f36bb3cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.673+0000 7f36bb3cf640 10 monclient(hunting): authenticate will time out at 1362.868163s 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.673+0000 7f36b916b640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f36b40a9d70 0x7f36b41dadf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:60882/0 (socket says 172.21.15.100:60882) 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.673+0000 7f36b916b640 1 -- 172.21.15.100:0/2430583461 learned_addr learned my addr 172.21.15.100:0/2430583461 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 client.?.objecter ms_handle_connect 0x7f36b40a9d70 2024-03-29T11:44:02.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 client.?.objecter ms_handle_connect 0x7f36b41457f0 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 client.?.objecter ms_handle_connect 0x7f36b413f650 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497471014 0 0) 0x7f36b41cd330 con 0x7f36b40a9d70 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3698004eb0 con 0x7f36b40a9d70 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3175120534 0 0) 0x7f3698004eb0 con 0x7f36b40a9d70 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36b41cd330 con 0x7f36b40a9d70 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f36b0001990 con 0x7f36b40a9d70 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:02.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): dump: 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178478416 0 0) 0x7f36b41db500 con 0x7f36b413f650 2024-03-29T11:44:02.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3698004eb0 con 0x7f36b413f650 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.674+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4149105530 0 0) 0x7f36b41cd330 con 0x7f36b40a9d70 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 >> v1:172.21.15.191:6789/0 conn(0x7f36b41457f0 legacy=0x7f36b41d7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 >> v1:172.21.15.100:6789/0 conn(0x7f36b413f650 legacy=0x7f36b4137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36a9ffb640 1 monclient: found mon.c 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36a9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4138450 con 0x7f36b40a9d70 2024-03-29T11:44:02.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36a9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36bb3cf640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36bb3cf640 10 monclient: _renew_subs 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36bb3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2430583461 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b41386a0 con 0x7f36b40a9d70 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36bb3cf640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36bb3cf640 10 monclient: _renew_subs 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36bb3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2430583461 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b413bad0 con 0x7f36b40a9d70 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.675+0000 7f36bb3cf640 1 librados: init done 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.676+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36b0003610 con 0x7f36b40a9d70 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.676+0000 7f36a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.676+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f36b0003a20 con 0x7f36b40a9d70 2024-03-29T11:44:02.691 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.676+0000 7f36a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.676+0000 7f36a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.676+0000 7f36ab7fe640 4 set_mon_vals no callback set 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.676+0000 7f36a9ffb640 10 monclient: dump: 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:02.692 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.677+0000 7f36bb3cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.677+0000 7f36bb3cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.677+0000 7f36bb3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.678+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f36b0003b80 con 0x7f36b40a9d70 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.677+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2430583461 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3694005080 con 0x7f36b40a9d70 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.679+0000 7f36b896a640 10 monclient: get_auth_request con 0x7f369806ddc0 auth_method 0 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.679+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f36b00548a0 con 0x7f36b40a9d70 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.679+0000 7f36a9ffb640 10 client.4556.objecter ms_dispatch 0x7f36b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.679+0000 7f36a9ffb640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.679+0000 7f36a9ffb640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.679+0000 7f36a9ffb640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:44:02.693 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.679+0000 7f36a9ffb640 10 client.4556.objecter ms_handle_connect 0x7f369806ddc0 2024-03-29T11:44:02.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.693+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36b0050b40 con 0x7f36b40a9d70 2024-03-29T11:44:02.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.693+0000 7f36a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:02.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:02.693+0000 7f36a9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:03.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.043+0000 7f36bb3cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:03.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.043+0000 7f36bb3cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:03.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.043+0000 7f36bb3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:03.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.043+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2430583461 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3694005330 con 0x7f36b40a9d70 2024-03-29T11:44:03.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.044+0000 7f36a9ffb640 1 -- 172.21.15.100:0/2430583461 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f36b0050b40 con 0x7f36b40a9d70 2024-03-29T11:44:03.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.044+0000 7f36a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:03.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.044+0000 7f36a9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:03.049 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:44:03.049 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:03.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 10 librados: watch_flush enter 2024-03-29T11:44:03.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 10 librados: watch_flush exit 2024-03-29T11:44:03.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:44:03.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:44:03.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2430583461 >> v1:172.21.15.100:6816/3220289657 conn(0x7f369806ddc0 legacy=0x7f3698070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:03.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 10 monclient: shutdown 2024-03-29T11:44:03.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:03.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2430583461 >> v1:172.21.15.100:6790/0 conn(0x7f36b40a9d70 legacy=0x7f36b41dadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:03.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2430583461 shutdown_connections 2024-03-29T11:44:03.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2430583461 >> 172.21.15.100:0/2430583461 conn(0x7f36b4066a00 msgr2=0x7f36b40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:03.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2430583461 shutdown_connections 2024-03-29T11:44:03.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 1 -- 172.21.15.100:0/2430583461 wait complete. 2024-03-29T11:44:03.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:03.049+0000 7f36bb3cf640 1 librados: shutdown 2024-03-29T11:44:03.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:09.874 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:10.036 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.032+0000 7f14729c5640 10 monclient: get_monmap_and_config 2024-03-29T11:44:10.036 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.032+0000 7f14729c5640 10 monclient: build_initial_monmap 2024-03-29T11:44:10.036 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.032+0000 7f14729c5640 10 monclient: monmap: 2024-03-29T11:44:10.036 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:10.036 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:10.037 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:44:10.033605+0000 2024-03-29T11:44:10.037 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:44:10.033605+0000 2024-03-29T11:44:10.037 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:10.037 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:10.037 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:10.037 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:44:10.037 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:44:10.037 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:10.038 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.033+0000 7f14729c5640 1 Processor -- start 2024-03-29T11:44:10.038 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.033+0000 7f14729c5640 1 -- start start 2024-03-29T11:44:10.038 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.033+0000 7f14729c5640 10 monclient: init 2024-03-29T11:44:10.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.034+0000 7f14729c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:10.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.034+0000 7f14729c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:10.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.034+0000 7f14729c5640 10 monclient(hunting): picked mon.noname-a con 0x7f1464038c60 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:10.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.034+0000 7f14729c5640 10 monclient(hunting): picked mon.noname-b con 0x7f14640324b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:10.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14729c5640 10 monclient(hunting): picked mon.noname-c con 0x7f14640d11e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:10.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14729c5640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640caf80 con 0x7f1464038c60 2024-03-29T11:44:10.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14711c2640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f14640324b0 0x7f14640cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:51746/0 (socket says 172.21.15.100:51746) 2024-03-29T11:44:10.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14711c2640 1 -- 172.21.15.100:0/1490517223 learned_addr learned my addr 172.21.15.100:0/1490517223 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:10.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14729c5640 1 -- 172.21.15.100:0/1490517223 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1464039850 con 0x7f14640d11e0 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14729c5640 1 -- 172.21.15.100:0/1490517223 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640c78d0 con 0x7f14640324b0 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14729c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14729c5640 10 monclient(hunting): authenticate will time out at 1370.230418s 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461173278 0 0) 0x7f1464039850 con 0x7f14640d11e0 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14709c1640 10 monclient(hunting): _negotiate 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14709c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14709c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14709c1640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.035+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1458003520 con 0x7f14640d11e0 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741125083 0 0) 0x7f14640c78d0 con 0x7f14640324b0 2024-03-29T11:44:10.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient(hunting): _negotiate 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1464039850 con 0x7f14640324b0 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771987053 0 0) 0x7f14640caf80 con 0x7f1464038c60 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient(hunting): _negotiate 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14640c78d0 con 0x7f1464038c60 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411794368 0 0) 0x7f1458003520 con 0x7f14640d11e0 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 >> v1:172.21.15.100:6790/0 conn(0x7f14640324b0 legacy=0x7f14640cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 >> v1:172.21.15.100:6789/0 conn(0x7f1464038c60 legacy=0x7f14640c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient: _finish_hunting 0 2024-03-29T11:44:10.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 1 monclient: found mon.noname-c 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640c7a60 con 0x7f14640d11e0 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient: _finish_auth 0 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient: _check_auth_tickets 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f146c04f730 con 0x7f14640d11e0 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14709c1640 10 monclient: dump: 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:10.046 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14729c5640 5 monclient: authenticate success, global_id 4441 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.036+0000 7f14729c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.037+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f146c075f10 con 0x7f14640d11e0 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.037+0000 7f14709c1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.037+0000 7f14709c1640 1 -- 172.21.15.100:0/1490517223 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f146c0758b0 con 0x7f14640d11e0 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.037+0000 7f14709c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.037+0000 7f14709c1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.037+0000 7f14709c1640 10 monclient: dump: 2024-03-29T11:44:10.047 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.037+0000 7f14729c5640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.037+0000 7f14729c5640 4 set_mon_vals no callback set 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.038+0000 7f14729c5640 10 monclient: shutdown 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.038+0000 7f14729c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.038+0000 7f14729c5640 1 -- 172.21.15.100:0/1490517223 >> v1:172.21.15.191:6789/0 conn(0x7f14640d11e0 legacy=0x7f14640d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.038+0000 7f14729c5640 1 -- 172.21.15.100:0/1490517223 shutdown_connections 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.038+0000 7f14729c5640 1 -- 172.21.15.100:0/1490517223 >> 172.21.15.100:0/1490517223 conn(0x7f1464005c20 msgr2=0x7f1464006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.038+0000 7f14729c5640 1 -- 172.21.15.100:0/1490517223 shutdown_connections 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.038+0000 7f14729c5640 1 -- 172.21.15.100:0/1490517223 wait complete. 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.039+0000 7f14729c5640 10 monclient: build_initial_monmap 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.039+0000 7f14729c5640 10 monclient: monmap: 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:44:10.049 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:44:10.050 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:10.052 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.039+0000 7f14729c5640 1 librados: starting msgr at 2024-03-29T11:44:10.052 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.039+0000 7f14729c5640 1 librados: starting objecter 2024-03-29T11:44:10.052 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.040+0000 7f14729c5640 1 Processor -- start 2024-03-29T11:44:10.052 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.040+0000 7f14729c5640 1 -- start start 2024-03-29T11:44:10.052 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.040+0000 7f14729c5640 1 librados: setting wanted keys 2024-03-29T11:44:10.052 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.040+0000 7f14729c5640 1 librados: calling monclient init 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.040+0000 7f14729c5640 10 monclient: init 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14729c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14729c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14729c5640 10 monclient(hunting): picked mon.noname-c con 0x7f14640324b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14729c5640 10 monclient(hunting): picked mon.noname-a con 0x7f1464038c60 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14729c5640 10 monclient(hunting): picked mon.noname-b con 0x7f14640d11e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14729c5640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640bac10 con 0x7f1464038c60 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14711c2640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f1464038c60 0x7f14640b9df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:44822/0 (socket says 172.21.15.100:44822) 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146402e2f0 con 0x7f14640d11e0 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14711c2640 1 -- 172.21.15.100:0/2170999768 learned_addr learned my addr 172.21.15.100:0/2170999768 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 10 client.?.objecter ms_handle_connect 0x7f14640d11e0 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 10 client.?.objecter ms_handle_connect 0x7f14640324b0 2024-03-29T11:44:10.053 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 10 client.?.objecter ms_handle_connect 0x7f1464038c60 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640d43a0 con 0x7f14640324b0 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14729c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14729c5640 10 monclient(hunting): authenticate will time out at 1370.236815s 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621590931 0 0) 0x7f14640d43a0 con 0x7f14640324b0 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.041+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1450004eb0 con 0x7f14640324b0 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569432823 0 0) 0x7f146402e2f0 con 0x7f14640d11e0 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:10.054 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1450006230 con 0x7f14640d11e0 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576146375 0 0) 0x7f14640bac10 con 0x7f1464038c60 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f146402e2f0 con 0x7f1464038c60 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 428849186 0 0) 0x7f1450006230 con 0x7f14640d11e0 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1450006830 con 0x7f14640d11e0 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1879078616 0 0) 0x7f1450004eb0 con 0x7f14640324b0 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1450006230 con 0x7f14640324b0 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f146c04f400 con 0x7f14640d11e0 2024-03-29T11:44:10.055 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): dump: 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1468001830 con 0x7f14640324b0 2024-03-29T11:44:10.056 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient(hunting): dump: 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 864743879 0 0) 0x7f1450006230 con 0x7f14640324b0 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 >> v1:172.21.15.191:6789/0 conn(0x7f14640d11e0 legacy=0x7f14640ba500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.057 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 >> v1:172.21.15.100:6789/0 conn(0x7f1464038c60 legacy=0x7f14640b9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 monclient: found mon.c 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640d4530 con 0x7f14640324b0 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14729c5640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14729c5640 10 monclient: _renew_subs 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14729c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.042+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146402d170 con 0x7f14640324b0 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14729c5640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:10.058 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14729c5640 10 monclient: _renew_subs 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14729c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1464164e10 con 0x7f14640324b0 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14729c5640 1 librados: init done 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1468003610 con 0x7f14640324b0 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1468003920 con 0x7f14640324b0 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14627fc640 10 monclient: dump: 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:10.059 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.043+0000 7f14637fe640 4 set_mon_vals no callback set 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.044+0000 7f14729c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.044+0000 7f14729c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.044+0000 7f14729c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.044+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1454005080 con 0x7f14640324b0 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.045+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f14680198f0 con 0x7f14640324b0 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.045+0000 7f14711c2640 10 monclient: get_auth_request con 0x7f145006e460 auth_method 0 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.045+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f1468083510 con 0x7f14640324b0 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.045+0000 7f14627fc640 10 client.4562.objecter ms_dispatch 0x7f146c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:10.060 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.045+0000 7f14627fc640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:10.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.045+0000 7f14627fc640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:10.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.045+0000 7f14627fc640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:44:10.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.046+0000 7f14627fc640 10 client.4562.objecter ms_handle_connect 0x7f145006e460 2024-03-29T11:44:10.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.055+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f146804ee50 con 0x7f14640324b0 2024-03-29T11:44:10.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.055+0000 7f14627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:10.061 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.055+0000 7f14627fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:10.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.422+0000 7f14729c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:10.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.422+0000 7f14729c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:10.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.422+0000 7f14729c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:10.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.422+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1454005330 con 0x7f14640324b0 2024-03-29T11:44:10.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.424+0000 7f14627fc640 1 -- 172.21.15.100:0/2170999768 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f1468082bc0 con 0x7f14640324b0 2024-03-29T11:44:10.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.424+0000 7f14627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:10.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.424+0000 7f14627fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:10.428 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:44:10.429 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:10.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 10 librados: watch_flush enter 2024-03-29T11:44:10.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 10 librados: watch_flush exit 2024-03-29T11:44:10.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:44:10.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:44:10.432 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 >> v1:172.21.15.100:6816/3220289657 conn(0x7f145006e460 legacy=0x7f1450070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 10 monclient: shutdown 2024-03-29T11:44:10.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:10.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 >> v1:172.21.15.100:6790/0 conn(0x7f14640324b0 legacy=0x7f1464164520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:10.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 shutdown_connections 2024-03-29T11:44:10.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 >> 172.21.15.100:0/2170999768 conn(0x7f1464005c20 msgr2=0x7f14640c64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:10.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 shutdown_connections 2024-03-29T11:44:10.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 1 -- 172.21.15.100:0/2170999768 wait complete. 2024-03-29T11:44:10.433 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:10.428+0000 7f14729c5640 1 librados: shutdown 2024-03-29T11:44:10.446 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:17.253 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:17.404 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.399+0000 7f73525f6640 10 monclient: get_monmap_and_config 2024-03-29T11:44:17.404 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.399+0000 7f73525f6640 10 monclient: build_initial_monmap 2024-03-29T11:44:17.404 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.400+0000 7f73525f6640 10 monclient: monmap: 2024-03-29T11:44:17.404 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:17.404 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:17.404 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:44:17.401030+0000 2024-03-29T11:44:17.404 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:44:17.401030+0000 2024-03-29T11:44:17.404 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:17.404 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:17.405 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:17.405 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:44:17.405 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:44:17.405 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:17.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.401+0000 7f73525f6640 1 Processor -- start 2024-03-29T11:44:17.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.401+0000 7f73525f6640 1 -- start start 2024-03-29T11:44:17.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.401+0000 7f73525f6640 10 monclient: init 2024-03-29T11:44:17.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.402+0000 7f73525f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:17.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.402+0000 7f73525f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:17.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.402+0000 7f73525f6640 10 monclient(hunting): picked mon.noname-b con 0x7f734c0b4330 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:17.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.402+0000 7f73525f6640 10 monclient(hunting): picked mon.noname-c con 0x7f734c1470b0 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:17.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.402+0000 7f73525f6640 10 monclient(hunting): picked mon.noname-a con 0x7f734c1439b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:17.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.402+0000 7f73525f6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c06f2a0 con 0x7f734c1439b0 2024-03-29T11:44:17.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.402+0000 7f73525f6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c146b90 con 0x7f734c1470b0 2024-03-29T11:44:17.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.402+0000 7f73525f6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0b2ca0 con 0x7f734c0b4330 2024-03-29T11:44:17.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.402+0000 7f73525f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:17.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.402+0000 7f73525f6640 10 monclient(hunting): authenticate will time out at 1377.597759s 2024-03-29T11:44:17.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.407+0000 7f734bfff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f734c0b4330 0x7f734c0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:47837/0 (socket says 172.21.15.100:47837) 2024-03-29T11:44:17.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.407+0000 7f734bfff640 1 -- 172.21.15.100:0/1926149571 learned_addr learned my addr 172.21.15.100:0/1926149571 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:17.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 1 -- 172.21.15.100:0/1926149571 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714378249 0 0) 0x7f734c0b2ca0 con 0x7f734c0b4330 2024-03-29T11:44:17.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:17.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:17.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 1 -- 172.21.15.100:0/1926149571 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7334003520 con 0x7f734c0b4330 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 1 -- 172.21.15.100:0/1926149571 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 644992930 0 0) 0x7f7334003520 con 0x7f734c0b4330 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 1 -- 172.21.15.100:0/1926149571 >> v1:172.21.15.191:6789/0 conn(0x7f734c1470b0 legacy=0x7f734c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 1 -- 172.21.15.100:0/1926149571 >> v1:172.21.15.100:6789/0 conn(0x7f734c1439b0 legacy=0x7f734c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 1 monclient: found mon.noname-b 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 1 -- 172.21.15.100:0/1926149571 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c06e0c0 con 0x7f734c0b4330 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient: _finish_auth 0 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:17.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f73525f6640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f73525f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 1 -- 172.21.15.100:0/1926149571 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f733c001670 con 0x7f734c0b4330 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient: dump: 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:17.415 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:17.416 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:17.416 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:17.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f73525f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:17.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 1 -- 172.21.15.100:0/1926149571 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f733c001900 con 0x7f734c0b4330 2024-03-29T11:44:17.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f734affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:17.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f73525f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:17.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.409+0000 7f73525f6640 4 set_mon_vals no callback set 2024-03-29T11:44:17.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.410+0000 7f73525f6640 10 monclient: shutdown 2024-03-29T11:44:17.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.410+0000 7f73525f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:17.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.410+0000 7f73525f6640 1 -- 172.21.15.100:0/1926149571 >> v1:172.21.15.100:6790/0 conn(0x7f734c0b4330 legacy=0x7f734c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:17.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.410+0000 7f734affd640 1 -- 172.21.15.100:0/1926149571 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f733c001670 con 0x7f734c0b4330 2024-03-29T11:44:17.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.410+0000 7f734affd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:44:17.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.413+0000 7f73525f6640 1 -- 172.21.15.100:0/1926149571 shutdown_connections 2024-03-29T11:44:17.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.413+0000 7f73525f6640 1 -- 172.21.15.100:0/1926149571 >> 172.21.15.100:0/1926149571 conn(0x7f734c066a00 msgr2=0x7f734c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:17.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.413+0000 7f73525f6640 1 -- 172.21.15.100:0/1926149571 shutdown_connections 2024-03-29T11:44:17.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.413+0000 7f73525f6640 1 -- 172.21.15.100:0/1926149571 wait complete. 2024-03-29T11:44:17.419 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.414+0000 7f73525f6640 10 monclient: build_initial_monmap 2024-03-29T11:44:17.419 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.414+0000 7f73525f6640 10 monclient: monmap: 2024-03-29T11:44:17.419 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:17.419 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:17.419 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:44:17.419 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:44:17.419 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.415+0000 7f73525f6640 1 librados: starting msgr at 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.415+0000 7f73525f6640 1 librados: starting objecter 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.415+0000 7f73525f6640 1 Processor -- start 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.415+0000 7f73525f6640 1 -- start start 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.415+0000 7f73525f6640 1 librados: setting wanted keys 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.415+0000 7f73525f6640 1 librados: calling monclient init 2024-03-29T11:44:17.420 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.415+0000 7f73525f6640 10 monclient: init 2024-03-29T11:44:17.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.416+0000 7f73525f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:17.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.416+0000 7f73525f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:17.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.416+0000 7f73525f6640 10 monclient(hunting): picked mon.noname-b con 0x7f734c0b4330 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:17.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.419+0000 7f73525f6640 10 monclient(hunting): picked mon.noname-c con 0x7f734c1439b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:17.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.419+0000 7f73525f6640 10 monclient(hunting): picked mon.noname-a con 0x7f734c1470b0 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:17.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.419+0000 7f73525f6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0ada50 con 0x7f734c1470b0 2024-03-29T11:44:17.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.419+0000 7f73525f6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0adca0 con 0x7f734c0b4330 2024-03-29T11:44:17.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.419+0000 7f73525f6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734c0ac700 con 0x7f734c1439b0 2024-03-29T11:44:17.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.419+0000 7f73525f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:17.423 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.419+0000 7f73525f6640 10 monclient(hunting): authenticate will time out at 1377.614426s 2024-03-29T11:44:17.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.422+0000 7f734bfff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f734c0b4330 0x7f734c1e3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:41355/0 (socket says 172.21.15.100:41355) 2024-03-29T11:44:17.426 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.423+0000 7f734bfff640 1 -- 172.21.15.100:0/1937122115 learned_addr learned my addr 172.21.15.100:0/1937122115 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:17.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.423+0000 7f7348ff9640 10 client.?.objecter ms_handle_connect 0x7f734c0b4330 2024-03-29T11:44:17.427 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.423+0000 7f7348ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:17.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.424+0000 7f7348ff9640 10 client.?.objecter ms_handle_connect 0x7f734c1470b0 2024-03-29T11:44:17.428 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.424+0000 7f7348ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:17.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.424+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080612621 0 0) 0x7f734c0adca0 con 0x7f734c0b4330 2024-03-29T11:44:17.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.424+0000 7f7348ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:17.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.424+0000 7f7348ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:17.429 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:17.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:44:17.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7338004eb0 con 0x7f734c0b4330 2024-03-29T11:44:17.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 10 client.?.objecter ms_handle_connect 0x7f734c1439b0 2024-03-29T11:44:17.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:17.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449462632 0 0) 0x7f734c0ada50 con 0x7f734c1470b0 2024-03-29T11:44:17.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:17.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:17.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:17.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:44:17.430 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.425+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f734c0adca0 con 0x7f734c1470b0 2024-03-29T11:44:17.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.428+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 19180648 0 0) 0x7f734c0adca0 con 0x7f734c1470b0 2024-03-29T11:44:17.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.428+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f734c0ada50 con 0x7f734c1470b0 2024-03-29T11:44:17.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.428+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 141982355 0 0) 0x7f7338004eb0 con 0x7f734c0b4330 2024-03-29T11:44:17.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.428+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f734c0adca0 con 0x7f734c0b4330 2024-03-29T11:44:17.435 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.428+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650042697 0 0) 0x7f734c0ac700 con 0x7f734c1439b0 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.428+0000 7f7348ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.428+0000 7f7348ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.428+0000 7f7348ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.428+0000 7f7348ff9640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.428+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7338004eb0 con 0x7f734c1439b0 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f7344001b00 con 0x7f734c1470b0 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:17.436 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f733c001630 con 0x7f734c0b4330 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:17.437 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2333310736 0 0) 0x7f734c0adca0 con 0x7f734c0b4330 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 >> v1:172.21.15.100:6790/0 conn(0x7f734c1439b0 legacy=0x7f734c0b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 >> v1:172.21.15.100:6789/0 conn(0x7f734c1470b0 legacy=0x7f734c1ef650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 1 monclient: found mon.b 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c1e4520 con 0x7f734c0b4330 2024-03-29T11:44:17.438 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f7348ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f73525f6640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f73525f6640 10 monclient: _renew_subs 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f73525f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f73525f6640 1 -- 172.21.15.100:0/1937122115 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734c0ab8b0 con 0x7f734c0b4330 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f73525f6640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f73525f6640 10 monclient: _renew_subs 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f73525f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f73525f6640 1 -- 172.21.15.100:0/1937122115 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f734c0ac3c0 con 0x7f734c0b4330 2024-03-29T11:44:17.439 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.429+0000 7f73525f6640 1 librados: init done 2024-03-29T11:44:17.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.432+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f733c004a90 con 0x7f734c0b4330 2024-03-29T11:44:17.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.432+0000 7f7348ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:17.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.432+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f733c004da0 con 0x7f734c0b4330 2024-03-29T11:44:17.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.432+0000 7f7348ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:17.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.432+0000 7f7348ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.432+0000 7f7348ff9640 10 monclient: dump: 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.432+0000 7f7349ffb640 4 set_mon_vals no callback set 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.432+0000 7f73525f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:17.442 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.433+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f733c01ad70 con 0x7f734c0b4330 2024-03-29T11:44:17.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.433+0000 7f73525f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:17.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.433+0000 7f73525f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:44:17.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.433+0000 7f73525f6640 1 -- 172.21.15.100:0/1937122115 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732c005080 con 0x7f734c0b4330 2024-03-29T11:44:17.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.434+0000 7f734b7fe640 10 monclient: get_auth_request con 0x7f733806dd10 auth_method 0 2024-03-29T11:44:17.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.434+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f733c084ac0 con 0x7f734c0b4330 2024-03-29T11:44:17.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.434+0000 7f7348ff9640 10 client.4447.objecter ms_dispatch 0x7f734c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:17.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.434+0000 7f7348ff9640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:17.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.434+0000 7f7348ff9640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:17.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.434+0000 7f7348ff9640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-29T11:44:17.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.434+0000 7f7348ff9640 10 client.4447.objecter ms_handle_connect 0x7f733806dd10 2024-03-29T11:44:17.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.441+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f733c0503a0 con 0x7f734c0b4330 2024-03-29T11:44:17.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.441+0000 7f7348ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:17.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.441+0000 7f7348ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:17.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.763+0000 7f73525f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:17.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.763+0000 7f73525f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:17.767 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.763+0000 7f73525f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:44:17.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.763+0000 7f73525f6640 1 -- 172.21.15.100:0/1937122115 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f732c005330 con 0x7f734c0b4330 2024-03-29T11:44:17.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.765+0000 7f7348ff9640 1 -- 172.21.15.100:0/1937122115 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f733c0503a0 con 0x7f734c0b4330 2024-03-29T11:44:17.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.765+0000 7f7348ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:17.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.765+0000 7f7348ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:17.771 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:44:17.771 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:17.772 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.768+0000 7f73525f6640 10 librados: watch_flush enter 2024-03-29T11:44:17.772 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.768+0000 7f73525f6640 10 librados: watch_flush exit 2024-03-29T11:44:17.773 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.768+0000 7f73525f6640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-29T11:44:17.773 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.768+0000 7f73525f6640 10 client.4447.objecter successfully canceled tick 2024-03-29T11:44:17.774 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.769+0000 7f73525f6640 1 -- 172.21.15.100:0/1937122115 >> v1:172.21.15.100:6816/3220289657 conn(0x7f733806dd10 legacy=0x7f73380701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:17.774 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.769+0000 7f73525f6640 10 monclient: shutdown 2024-03-29T11:44:17.774 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.769+0000 7f73525f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:17.774 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.769+0000 7f73525f6640 1 -- 172.21.15.100:0/1937122115 >> v1:172.21.15.191:6789/0 conn(0x7f734c0b4330 legacy=0x7f734c1e3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:17.774 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.770+0000 7f73525f6640 1 -- 172.21.15.100:0/1937122115 shutdown_connections 2024-03-29T11:44:17.775 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.770+0000 7f73525f6640 1 -- 172.21.15.100:0/1937122115 >> 172.21.15.100:0/1937122115 conn(0x7f734c066a00 msgr2=0x7f734c0b82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:17.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.771+0000 7f73525f6640 1 -- 172.21.15.100:0/1937122115 shutdown_connections 2024-03-29T11:44:17.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.773+0000 7f73525f6640 1 -- 172.21.15.100:0/1937122115 wait complete. 2024-03-29T11:44:17.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:17.773+0000 7f73525f6640 1 librados: shutdown 2024-03-29T11:44:17.796 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:24.603 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:24.753 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.747+0000 7ff889777640 10 monclient: get_monmap_and_config 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.747+0000 7ff889777640 10 monclient: build_initial_monmap 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.747+0000 7ff889777640 10 monclient: monmap: 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:44:24.748453+0000 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:44:24.748453+0000 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:24.754 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.749+0000 7ff889777640 1 Processor -- start 2024-03-29T11:44:24.755 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.749+0000 7ff889777640 1 -- start start 2024-03-29T11:44:24.755 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.749+0000 7ff889777640 10 monclient: init 2024-03-29T11:44:24.755 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff889777640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:24.755 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff889777640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:24.755 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff889777640 10 monclient(hunting): picked mon.noname-a con 0x7ff8840a9930 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:24.755 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff889777640 10 monclient(hunting): picked mon.noname-b con 0x7ff884149dc0 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff889777640 10 monclient(hunting): picked mon.noname-c con 0x7ff88414ff40 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff889777640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840aae60 con 0x7ff8840a9930 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff889777640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840b4310 con 0x7ff88414ff40 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff889777640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840afdd0 con 0x7ff884149dc0 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff889777640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff889777640 10 monclient(hunting): authenticate will time out at 1384.945481s 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff8827fc640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7ff884149dc0 0x7ff88414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:45454/0 (socket says 172.21.15.100:45454) 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.750+0000 7ff8827fc640 1 -- 172.21.15.100:0/2944087587 learned_addr learned my addr 172.21.15.100:0/2944087587 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.751+0000 7ff881ffb640 1 -- 172.21.15.100:0/2944087587 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082450267 0 0) 0x7ff8840afdd0 con 0x7ff884149dc0 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.751+0000 7ff881ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.751+0000 7ff881ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.751+0000 7ff881ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.751+0000 7ff881ffb640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:44:24.756 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.751+0000 7ff881ffb640 1 -- 172.21.15.100:0/2944087587 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff86c003520 con 0x7ff884149dc0 2024-03-29T11:44:24.757 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.753+0000 7ff881ffb640 1 -- 172.21.15.100:0/2944087587 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938559892 0 0) 0x7ff86c003520 con 0x7ff884149dc0 2024-03-29T11:44:24.757 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.753+0000 7ff881ffb640 1 -- 172.21.15.100:0/2944087587 >> v1:172.21.15.191:6789/0 conn(0x7ff88414ff40 legacy=0x7ff884152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:24.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.754+0000 7ff881ffb640 1 -- 172.21.15.100:0/2944087587 >> v1:172.21.15.100:6789/0 conn(0x7ff8840a9930 legacy=0x7ff884143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:24.760 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 1 monclient: found mon.noname-b 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 1 -- 172.21.15.100:0/2944087587 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88414a950 con 0x7ff884149dc0 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: _finish_auth 0 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 1 -- 172.21.15.100:0/2944087587 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff87c001a10 con 0x7ff884149dc0 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: dump: 2024-03-29T11:44:24.761 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff889777640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff889777640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 1 -- 172.21.15.100:0/2944087587 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff87c001be0 con 0x7ff884149dc0 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:24.762 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 1 -- 172.21.15.100:0/2944087587 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff87c002030 con 0x7ff884149dc0 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.755+0000 7ff881ffb640 10 monclient: dump: 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:24.763 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:24.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.757+0000 7ff889777640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:24.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.757+0000 7ff889777640 4 set_mon_vals no callback set 2024-03-29T11:44:24.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.758+0000 7ff889777640 10 monclient: shutdown 2024-03-29T11:44:24.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.758+0000 7ff889777640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:24.764 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.758+0000 7ff889777640 1 -- 172.21.15.100:0/2944087587 >> v1:172.21.15.100:6790/0 conn(0x7ff884149dc0 legacy=0x7ff88414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:24.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.761+0000 7ff889777640 1 -- 172.21.15.100:0/2944087587 shutdown_connections 2024-03-29T11:44:24.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.761+0000 7ff889777640 1 -- 172.21.15.100:0/2944087587 >> 172.21.15.100:0/2944087587 conn(0x7ff8840b35e0 msgr2=0x7ff8840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:24.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.761+0000 7ff889777640 1 -- 172.21.15.100:0/2944087587 shutdown_connections 2024-03-29T11:44:24.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.761+0000 7ff889777640 1 -- 172.21.15.100:0/2944087587 wait complete. 2024-03-29T11:44:24.766 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.762+0000 7ff889777640 10 monclient: build_initial_monmap 2024-03-29T11:44:24.768 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.762+0000 7ff889777640 10 monclient: monmap: 2024-03-29T11:44:24.768 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:24.768 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:24.768 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:44:24.768 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:44:24.768 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:24.768 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:24.768 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:24.768 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.763+0000 7ff889777640 1 librados: starting msgr at 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.763+0000 7ff889777640 1 librados: starting objecter 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.763+0000 7ff889777640 1 Processor -- start 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.763+0000 7ff889777640 1 -- start start 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.763+0000 7ff889777640 1 librados: setting wanted keys 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.763+0000 7ff889777640 1 librados: calling monclient init 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.763+0000 7ff889777640 10 monclient: init 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.764+0000 7ff889777640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.764+0000 7ff889777640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.764+0000 7ff889777640 10 monclient(hunting): picked mon.noname-c con 0x7ff8840a9930 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:24.769 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.764+0000 7ff889777640 10 monclient(hunting): picked mon.noname-a con 0x7ff884149dc0 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:24.770 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.764+0000 7ff889777640 10 monclient(hunting): picked mon.noname-b con 0x7ff88414ff40 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:24.770 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.764+0000 7ff889777640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff884149b90 con 0x7ff884149dc0 2024-03-29T11:44:24.770 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.764+0000 7ff889777640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff884071e30 con 0x7ff88414ff40 2024-03-29T11:44:24.770 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.764+0000 7ff889777640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff884142be0 con 0x7ff8840a9930 2024-03-29T11:44:24.770 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.764+0000 7ff889777640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:24.770 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.764+0000 7ff889777640 10 monclient(hunting): authenticate will time out at 1384.959339s 2024-03-29T11:44:24.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.770+0000 7ff8827fc640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff884149dc0 0x7ff8841424d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:44383/0 (socket says 172.21.15.100:44383) 2024-03-29T11:44:24.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.770+0000 7ff8827fc640 1 -- 172.21.15.100:0/145410723 learned_addr learned my addr 172.21.15.100:0/145410723 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:24.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.770+0000 7ff86b7fe640 10 client.?.objecter ms_handle_connect 0x7ff884149dc0 2024-03-29T11:44:24.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.770+0000 7ff86b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:24.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.770+0000 7ff86b7fe640 10 client.?.objecter ms_handle_connect 0x7ff8840a9930 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.770+0000 7ff86b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728903331 0 0) 0x7ff88414a950 con 0x7ff884149dc0 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff870004eb0 con 0x7ff884149dc0 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421230479 0 0) 0x7ff884142be0 con 0x7ff8840a9930 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff88414a950 con 0x7ff8840a9930 2024-03-29T11:44:24.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2175596606 0 0) 0x7ff870004eb0 con 0x7ff884149dc0 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff884142be0 con 0x7ff884149dc0 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff87c001d50 con 0x7ff884149dc0 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.771+0000 7ff86b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 10 monclient(hunting): dump: 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:24.780 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1976455601 0 0) 0x7ff88414a950 con 0x7ff8840a9930 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff870004eb0 con 0x7ff8840a9930 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2276573616 0 0) 0x7ff884142be0 con 0x7ff884149dc0 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 >> v1:172.21.15.100:6790/0 conn(0x7ff8840a9930 legacy=0x7ff88406f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 >> v1:172.21.15.191:6789/0 conn(0x7ff88414ff40 legacy=0x7ff8841deb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 1 monclient: found mon.a 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8841df250 con 0x7ff884149dc0 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:24.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.772+0000 7ff86b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:24.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.773+0000 7ff86b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:24.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.773+0000 7ff889777640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:44:24.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.773+0000 7ff889777640 10 monclient: _renew_subs 2024-03-29T11:44:24.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.773+0000 7ff889777640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:24.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.773+0000 7ff889777640 1 -- 172.21.15.100:0/145410723 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff884072310 con 0x7ff884149dc0 2024-03-29T11:44:24.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.773+0000 7ff889777640 10 client.4485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:24.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.773+0000 7ff889777640 10 monclient: _renew_subs 2024-03-29T11:44:24.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.773+0000 7ff889777640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:24.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.773+0000 7ff889777640 1 -- 172.21.15.100:0/145410723 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88406fa10 con 0x7ff884149dc0 2024-03-29T11:44:24.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.773+0000 7ff889777640 1 librados: init done 2024-03-29T11:44:24.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.775+0000 7ff889777640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:24.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.775+0000 7ff889777640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:24.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.775+0000 7ff889777640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:24.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.775+0000 7ff889777640 1 -- 172.21.15.100:0/145410723 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8840726b0 con 0x7ff884149dc0 2024-03-29T11:44:24.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.778+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff87c0020d0 con 0x7ff884149dc0 2024-03-29T11:44:24.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.778+0000 7ff86b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:24.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.779+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff87c001eb0 con 0x7ff884149dc0 2024-03-29T11:44:24.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.779+0000 7ff86b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:24.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.779+0000 7ff86b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.779+0000 7ff86b7fe640 10 monclient: dump: 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.779+0000 7ff8817fa640 4 set_mon_vals no callback set 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.779+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7ff87c01ade0 con 0x7ff884149dc0 2024-03-29T11:44:24.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.780+0000 7ff882ffd640 10 monclient: get_auth_request con 0x7ff87006dd20 auth_method 0 2024-03-29T11:44:24.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.780+0000 7ff86b7fe640 10 client.4485.objecter ms_handle_connect 0x7ff87006dd20 2024-03-29T11:44:24.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.781+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7ff87c084c20 con 0x7ff884149dc0 2024-03-29T11:44:24.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.781+0000 7ff86b7fe640 10 client.4485.objecter ms_dispatch 0x7ff884000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:24.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.781+0000 7ff86b7fe640 3 client.4485.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:24.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.781+0000 7ff86b7fe640 3 client.4485.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:24.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.781+0000 7ff86b7fe640 20 client.4485.objecter dump_active .. 0 homeless 2024-03-29T11:44:24.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.784+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff87c050500 con 0x7ff884149dc0 2024-03-29T11:44:24.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.784+0000 7ff86b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:24.788 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:24.784+0000 7ff86b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:25.118 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.114+0000 7ff889777640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:25.118 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.114+0000 7ff889777640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:25.119 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.114+0000 7ff889777640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:25.119 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.114+0000 7ff889777640 1 -- 172.21.15.100:0/145410723 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff884075f10 con 0x7ff884149dc0 2024-03-29T11:44:25.119 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.115+0000 7ff86b7fe640 1 -- 172.21.15.100:0/145410723 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7ff87c052430 con 0x7ff884149dc0 2024-03-29T11:44:25.119 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.116+0000 7ff86b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:25.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.116+0000 7ff86b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:25.121 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:44:25.121 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.120+0000 7ff889777640 10 librados: watch_flush enter 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.120+0000 7ff889777640 10 librados: watch_flush exit 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.120+0000 7ff889777640 20 client.4485.objecter shutdown clearing up homeless session... 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.120+0000 7ff889777640 10 client.4485.objecter successfully canceled tick 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.121+0000 7ff889777640 1 -- 172.21.15.100:0/145410723 >> v1:172.21.15.100:6816/3220289657 conn(0x7ff87006dd20 legacy=0x7ff8700701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.121+0000 7ff889777640 10 monclient: shutdown 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.121+0000 7ff889777640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.121+0000 7ff889777640 1 -- 172.21.15.100:0/145410723 >> v1:172.21.15.100:6789/0 conn(0x7ff884149dc0 legacy=0x7ff8841424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.121+0000 7ff889777640 1 -- 172.21.15.100:0/145410723 shutdown_connections 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.121+0000 7ff889777640 1 -- 172.21.15.100:0/145410723 >> 172.21.15.100:0/145410723 conn(0x7ff8840b35e0 msgr2=0x7ff8840b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:25.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.121+0000 7ff889777640 1 -- 172.21.15.100:0/145410723 shutdown_connections 2024-03-29T11:44:25.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.121+0000 7ff889777640 1 -- 172.21.15.100:0/145410723 wait complete. 2024-03-29T11:44:25.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:25.122+0000 7ff889777640 1 librados: shutdown 2024-03-29T11:44:25.143 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:31.951 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:32.100 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.096+0000 7f698441e640 10 monclient: get_monmap_and_config 2024-03-29T11:44:32.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.096+0000 7f698441e640 10 monclient: build_initial_monmap 2024-03-29T11:44:32.101 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.096+0000 7f698441e640 10 monclient: monmap: 2024-03-29T11:44:32.101 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:32.101 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:32.101 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:44:32.097483+0000 2024-03-29T11:44:32.101 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:44:32.097483+0000 2024-03-29T11:44:32.101 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:32.101 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:32.102 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:32.102 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:44:32.102 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:44:32.102 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:32.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.097+0000 7f698441e640 1 Processor -- start 2024-03-29T11:44:32.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.097+0000 7f698441e640 1 -- start start 2024-03-29T11:44:32.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.097+0000 7f698441e640 10 monclient: init 2024-03-29T11:44:32.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.098+0000 7f698441e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:32.103 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.098+0000 7f698441e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:32.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.098+0000 7f698441e640 10 monclient(hunting): picked mon.noname-c con 0x7f697c0b0860 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:32.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.100+0000 7f698441e640 10 monclient(hunting): picked mon.noname-b con 0x7f697c13ef30 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:32.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.100+0000 7f698441e640 10 monclient(hunting): picked mon.noname-a con 0x7f697c0af8e0 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:32.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.100+0000 7f698441e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c0b0c40 con 0x7f697c0af8e0 2024-03-29T11:44:32.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.100+0000 7f698441e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c0aed10 con 0x7f697c0b0860 2024-03-29T11:44:32.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.100+0000 7f698441e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c0afd50 con 0x7f697c13ef30 2024-03-29T11:44:32.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.100+0000 7f698441e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:32.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.100+0000 7f698441e640 10 monclient(hunting): authenticate will time out at 1392.295985s 2024-03-29T11:44:32.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.101+0000 7f69819b9640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f697c13ef30 0x7f697c0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:40978/0 (socket says 172.21.15.100:40978) 2024-03-29T11:44:32.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.101+0000 7f69819b9640 1 -- 172.21.15.100:0/2983390 learned_addr learned my addr 172.21.15.100:0/2983390 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:32.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.101+0000 7f69811b8640 1 -- 172.21.15.100:0/2983390 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297756404 0 0) 0x7f697c0afd50 con 0x7f697c13ef30 2024-03-29T11:44:32.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.101+0000 7f69811b8640 10 monclient(hunting): _negotiate 2024-03-29T11:44:32.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.101+0000 7f69811b8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:32.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.101+0000 7f69811b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:32.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.102+0000 7f69811b8640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:44:32.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.102+0000 7f69811b8640 1 -- 172.21.15.100:0/2983390 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6968003520 con 0x7f697c13ef30 2024-03-29T11:44:32.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.102+0000 7f69811b8640 1 -- 172.21.15.100:0/2983390 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 506495225 0 0) 0x7f6968003520 con 0x7f697c13ef30 2024-03-29T11:44:32.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.102+0000 7f69811b8640 1 -- 172.21.15.100:0/2983390 >> v1:172.21.15.191:6789/0 conn(0x7f697c0b0860 legacy=0x7f697c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:32.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.102+0000 7f69811b8640 1 -- 172.21.15.100:0/2983390 >> v1:172.21.15.100:6789/0 conn(0x7f697c0af8e0 legacy=0x7f697c149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:32.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 10 monclient: _finish_hunting 0 2024-03-29T11:44:32.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 1 monclient: found mon.noname-b 2024-03-29T11:44:32.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:32.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:44:32.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 1 -- 172.21.15.100:0/2983390 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c0ab1b0 con 0x7f697c13ef30 2024-03-29T11:44:32.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 10 monclient: _finish_auth 0 2024-03-29T11:44:32.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 10 monclient: _check_auth_tickets 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 1 -- 172.21.15.100:0/2983390 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f696c001690 con 0x7f697c13ef30 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 10 monclient: dump: 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:32.111 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f698441e640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f698441e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 1 -- 172.21.15.100:0/2983390 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f696c001860 con 0x7f697c13ef30 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.104+0000 7f69811b8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.105+0000 7f698441e640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.105+0000 7f698441e640 4 set_mon_vals no callback set 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.105+0000 7f69811b8640 1 -- 172.21.15.100:0/2983390 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f696c001cb0 con 0x7f697c13ef30 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.105+0000 7f69811b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.105+0000 7f69811b8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.105+0000 7f69811b8640 10 monclient: dump: 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:32.112 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:32.113 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:32.113 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:32.113 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:32.113 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:32.113 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:32.113 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:32.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.106+0000 7f698441e640 10 monclient: shutdown 2024-03-29T11:44:32.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.106+0000 7f698441e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:32.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.106+0000 7f698441e640 1 -- 172.21.15.100:0/2983390 >> v1:172.21.15.100:6790/0 conn(0x7f697c13ef30 legacy=0x7f697c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:32.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.107+0000 7f698441e640 1 -- 172.21.15.100:0/2983390 shutdown_connections 2024-03-29T11:44:32.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.107+0000 7f698441e640 1 -- 172.21.15.100:0/2983390 >> 172.21.15.100:0/2983390 conn(0x7f697c06b660 msgr2=0x7f697c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:32.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.107+0000 7f698441e640 1 -- 172.21.15.100:0/2983390 shutdown_connections 2024-03-29T11:44:32.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.107+0000 7f698441e640 1 -- 172.21.15.100:0/2983390 wait complete. 2024-03-29T11:44:32.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.108+0000 7f698441e640 10 monclient: build_initial_monmap 2024-03-29T11:44:32.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.108+0000 7f698441e640 10 monclient: monmap: 2024-03-29T11:44:32.115 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:32.115 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.108+0000 7f698441e640 1 librados: starting msgr at 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.108+0000 7f698441e640 1 librados: starting objecter 2024-03-29T11:44:32.116 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.108+0000 7f698441e640 1 Processor -- start 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.108+0000 7f698441e640 1 -- start start 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.108+0000 7f698441e640 1 librados: setting wanted keys 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.108+0000 7f698441e640 1 librados: calling monclient init 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.108+0000 7f698441e640 10 monclient: init 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.109+0000 7f698441e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.109+0000 7f698441e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.109+0000 7f698441e640 10 monclient(hunting): picked mon.noname-b con 0x7f697c0af8e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.109+0000 7f698441e640 10 monclient(hunting): picked mon.noname-a con 0x7f697c0b0860 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.109+0000 7f698441e640 10 monclient(hunting): picked mon.noname-c con 0x7f697c13ef30 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.109+0000 7f698441e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c06f650 con 0x7f697c0b0860 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.109+0000 7f698441e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c1dec90 con 0x7f697c0af8e0 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.109+0000 7f698441e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c1df8c0 con 0x7f697c13ef30 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.109+0000 7f698441e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:32.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.109+0000 7f698441e640 10 monclient(hunting): authenticate will time out at 1392.304935s 2024-03-29T11:44:32.118 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.111+0000 7f69829bb640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f697c13ef30 0x7f697c1df1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:44608/0 (socket says 172.21.15.100:44608) 2024-03-29T11:44:32.118 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.111+0000 7f69829bb640 1 -- 172.21.15.100:0/1683002121 learned_addr learned my addr 172.21.15.100:0/1683002121 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:32.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.111+0000 7f6972ffd640 10 client.?.objecter ms_handle_connect 0x7f697c13ef30 2024-03-29T11:44:32.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.111+0000 7f6972ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:32.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.111+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981436156 0 0) 0x7f697c1df8c0 con 0x7f697c13ef30 2024-03-29T11:44:32.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.111+0000 7f6972ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:32.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.111+0000 7f6972ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:32.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.111+0000 7f6972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:32.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.111+0000 7f6972ffd640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.111+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6960004eb0 con 0x7f697c13ef30 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.112+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1305502079 0 0) 0x7f6960004eb0 con 0x7f697c13ef30 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.112+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f697c1df8c0 con 0x7f697c13ef30 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.112+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6974001de0 con 0x7f697c13ef30 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.112+0000 7f6972ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.112+0000 7f6972ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.112+0000 7f6972ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:32.124 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.112+0000 7f6972ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.112+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3294179021 0 0) 0x7f697c1df8c0 con 0x7f697c13ef30 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.113+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 >> v1:172.21.15.191:6789/0 conn(0x7f697c0af8e0 legacy=0x7f697c06e250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.113+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 >> v1:172.21.15.100:6789/0 conn(0x7f697c0b0860 legacy=0x7f697c06ef40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f6972ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:32.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f6972ffd640 1 monclient: found mon.c 2024-03-29T11:44:32.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f6972ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:32.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f6972ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:32.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c1dfa50 con 0x7f697c13ef30 2024-03-29T11:44:32.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f6972ffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:32.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f6972ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:32.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f6972ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:32.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f698441e640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:44:32.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f698441e640 10 monclient: _renew_subs 2024-03-29T11:44:32.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f698441e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:32.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f698441e640 1 -- 172.21.15.100:0/1683002121 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697c1e6e90 con 0x7f697c13ef30 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f698441e640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f698441e640 10 monclient: _renew_subs 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f698441e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f698441e640 1 -- 172.21.15.100:0/1683002121 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f697c1e7200 con 0x7f697c13ef30 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.114+0000 7f698441e640 1 librados: init done 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.115+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6974001f40 con 0x7f697c13ef30 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.115+0000 7f6972ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.115+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f6974003ca0 con 0x7f697c13ef30 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.115+0000 7f6972ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.115+0000 7f69809b7640 4 set_mon_vals no callback set 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.115+0000 7f6972ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.115+0000 7f6972ffd640 10 monclient: dump: 2024-03-29T11:44:32.127 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:32.128 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:32.128 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:32.128 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:32.128 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:32.128 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:32.129 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:32.129 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:32.129 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:32.129 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:32.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.116+0000 7f698441e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:32.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.116+0000 7f698441e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:32.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.116+0000 7f698441e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:32.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.116+0000 7f698441e640 1 -- 172.21.15.100:0/1683002121 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6964005080 con 0x7f697c13ef30 2024-03-29T11:44:32.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.117+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f6974003e00 con 0x7f697c13ef30 2024-03-29T11:44:32.130 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.118+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f6974084d10 con 0x7f697c13ef30 2024-03-29T11:44:32.130 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.119+0000 7f6972ffd640 10 client.4580.objecter ms_dispatch 0x7f697c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:32.130 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.119+0000 7f6972ffd640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:32.130 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.119+0000 7f6972ffd640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:32.130 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.119+0000 7f6972ffd640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:44:32.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.119+0000 7f69821ba640 10 monclient: get_auth_request con 0x7f696006dca0 auth_method 0 2024-03-29T11:44:32.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.120+0000 7f6972ffd640 10 client.4580.objecter ms_handle_connect 0x7f696006dca0 2024-03-29T11:44:32.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.126+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69740505f0 con 0x7f697c13ef30 2024-03-29T11:44:32.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.126+0000 7f6972ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:32.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.126+0000 7f6972ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:32.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.455+0000 7f698441e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:32.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.455+0000 7f698441e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:32.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.455+0000 7f698441e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:32.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.455+0000 7f698441e640 1 -- 172.21.15.100:0/1683002121 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6964005330 con 0x7f697c13ef30 2024-03-29T11:44:32.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.457+0000 7f6972ffd640 1 -- 172.21.15.100:0/1683002121 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f6974052520 con 0x7f697c13ef30 2024-03-29T11:44:32.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.457+0000 7f6972ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:32.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.457+0000 7f6972ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:32.462 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:44:32.462 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:32.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 10 librados: watch_flush enter 2024-03-29T11:44:32.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 10 librados: watch_flush exit 2024-03-29T11:44:32.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:44:32.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:44:32.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 1 -- 172.21.15.100:0/1683002121 >> v1:172.21.15.100:6816/3220289657 conn(0x7f696006dca0 legacy=0x7f6960070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:32.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 10 monclient: shutdown 2024-03-29T11:44:32.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:32.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 1 -- 172.21.15.100:0/1683002121 >> v1:172.21.15.100:6790/0 conn(0x7f697c13ef30 legacy=0x7f697c1df1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:32.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 1 -- 172.21.15.100:0/1683002121 shutdown_connections 2024-03-29T11:44:32.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 1 -- 172.21.15.100:0/1683002121 >> 172.21.15.100:0/1683002121 conn(0x7f697c06b660 msgr2=0x7f697c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:32.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 1 -- 172.21.15.100:0/1683002121 shutdown_connections 2024-03-29T11:44:32.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.461+0000 7f698441e640 1 -- 172.21.15.100:0/1683002121 wait complete. 2024-03-29T11:44:32.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:32.462+0000 7f698441e640 1 librados: shutdown 2024-03-29T11:44:32.482 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:39.290 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:39.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.436+0000 7f5fc759e640 10 monclient: get_monmap_and_config 2024-03-29T11:44:39.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.436+0000 7f5fc759e640 10 monclient: build_initial_monmap 2024-03-29T11:44:39.441 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.436+0000 7f5fc759e640 10 monclient: monmap: 2024-03-29T11:44:39.441 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:39.441 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:39.441 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:44:39.437644+0000 2024-03-29T11:44:39.441 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:44:39.437644+0000 2024-03-29T11:44:39.441 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:39.441 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:39.442 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:39.442 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:44:39.442 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:44:39.442 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:39.442 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.437+0000 7f5fc759e640 1 Processor -- start 2024-03-29T11:44:39.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.437+0000 7f5fc759e640 1 -- start start 2024-03-29T11:44:39.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.437+0000 7f5fc759e640 10 monclient: init 2024-03-29T11:44:39.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.438+0000 7f5fc759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:39.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.438+0000 7f5fc759e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:39.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.438+0000 7f5fc759e640 10 monclient(hunting): picked mon.noname-a con 0x7f5fc80b4920 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:39.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.438+0000 7f5fc759e640 10 monclient(hunting): picked mon.noname-b con 0x7f5fc80b4200 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:39.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.438+0000 7f5fc759e640 10 monclient(hunting): picked mon.noname-c con 0x7f5fc813f9a0 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:39.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.438+0000 7f5fc759e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc80b0930 con 0x7f5fc80b4920 2024-03-29T11:44:39.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.438+0000 7f5fc759e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc80af460 con 0x7f5fc813f9a0 2024-03-29T11:44:39.446 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.438+0000 7f5fc759e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc80aeda0 con 0x7f5fc80b4200 2024-03-29T11:44:39.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.438+0000 7f5fc759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:39.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.438+0000 7f5fc759e640 10 monclient(hunting): authenticate will time out at 1399.633584s 2024-03-29T11:44:39.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.441+0000 7f5fc659c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f5fc80b4920 0x7f5fc813f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:55073/0 (socket says 172.21.15.100:55073) 2024-03-29T11:44:39.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.441+0000 7f5fbdd9b640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5fc80b4200 0x7f5fc80b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:49822/0 (socket says 172.21.15.100:49822) 2024-03-29T11:44:39.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.441+0000 7f5fbdd9b640 1 -- 172.21.15.100:0/1583064641 learned_addr learned my addr 172.21.15.100:0/1583064641 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:39.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.442+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057114074 0 0) 0x7f5fc80b0930 con 0x7f5fc80b4920 2024-03-29T11:44:39.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:44:39.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:39.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:39.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:44:39.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb4003520 con 0x7f5fc80b4920 2024-03-29T11:44:39.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334544566 0 0) 0x7f5fc80aeda0 con 0x7f5fc80b4200 2024-03-29T11:44:39.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:44:39.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:39.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:39.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:44:39.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc80b0930 con 0x7f5fc80b4200 2024-03-29T11:44:39.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3636979032 0 0) 0x7f5fc80b0930 con 0x7f5fc80b4200 2024-03-29T11:44:39.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 >> v1:172.21.15.191:6789/0 conn(0x7f5fc813f9a0 legacy=0x7f5fc814d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:39.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.443+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 >> v1:172.21.15.100:6789/0 conn(0x7f5fc80b4920 legacy=0x7f5fc813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:39.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:44:39.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 1 monclient: found mon.noname-b 2024-03-29T11:44:39.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:39.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:44:39.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc80af980 con 0x7f5fc80b4200 2024-03-29T11:44:39.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:44:39.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:44:39.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:39.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc759e640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:44:39.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:39.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5fb00015f0 con 0x7f5fc80b4200 2024-03-29T11:44:39.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:39.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:39.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 10 monclient: dump: 2024-03-29T11:44:39.460 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:39.460 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:39.460 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:39.460 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:39.460 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:39.460 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:39.461 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:39.461 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:39.461 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:39.461 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:39.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:39.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fb0001750 con 0x7f5fc80b4200 2024-03-29T11:44:39.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:39.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:39.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.444+0000 7f5fc759e640 4 set_mon_vals no callback set 2024-03-29T11:44:39.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.445+0000 7f5fc5d9b640 1 -- 172.21.15.100:0/1583064641 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5fb0001b60 con 0x7f5fc80b4200 2024-03-29T11:44:39.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.445+0000 7f5fc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:39.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.445+0000 7f5fc5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:39.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.445+0000 7f5fc5d9b640 10 monclient: dump: 2024-03-29T11:44:39.463 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:39.463 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:39.463 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:39.463 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:39.463 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:39.464 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:39.464 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:39.464 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:39.464 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:39.464 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:39.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.445+0000 7f5fc759e640 10 monclient: shutdown 2024-03-29T11:44:39.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.445+0000 7f5fc759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:39.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.445+0000 7f5fc759e640 1 -- 172.21.15.100:0/1583064641 >> v1:172.21.15.100:6790/0 conn(0x7f5fc80b4200 legacy=0x7f5fc80b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:39.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.445+0000 7f5fc759e640 1 -- 172.21.15.100:0/1583064641 shutdown_connections 2024-03-29T11:44:39.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.445+0000 7f5fc759e640 1 -- 172.21.15.100:0/1583064641 >> 172.21.15.100:0/1583064641 conn(0x7f5fc806d230 msgr2=0x7f5fc8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:39.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.448+0000 7f5fc759e640 1 -- 172.21.15.100:0/1583064641 shutdown_connections 2024-03-29T11:44:39.479 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.452+0000 7f5fc759e640 1 -- 172.21.15.100:0/1583064641 wait complete. 2024-03-29T11:44:39.479 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.453+0000 7f5fc759e640 10 monclient: build_initial_monmap 2024-03-29T11:44:39.480 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.453+0000 7f5fc759e640 10 monclient: monmap: 2024-03-29T11:44:39.480 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:39.480 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:39.480 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:44:39.480 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:44:39.480 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:39.480 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:39.480 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:39.481 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:44:39.481 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:44:39.481 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:39.481 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.454+0000 7f5fc759e640 1 librados: starting msgr at 2024-03-29T11:44:39.481 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.454+0000 7f5fc759e640 1 librados: starting objecter 2024-03-29T11:44:39.481 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.454+0000 7f5fc759e640 1 Processor -- start 2024-03-29T11:44:39.482 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.455+0000 7f5fc759e640 1 -- start start 2024-03-29T11:44:39.482 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.455+0000 7f5fc759e640 1 librados: setting wanted keys 2024-03-29T11:44:39.482 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.455+0000 7f5fc759e640 1 librados: calling monclient init 2024-03-29T11:44:39.482 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.455+0000 7f5fc759e640 10 monclient: init 2024-03-29T11:44:39.482 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.456+0000 7f5fc759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:39.482 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.456+0000 7f5fc759e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:39.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.456+0000 7f5fc759e640 10 monclient(hunting): picked mon.noname-c con 0x7f5fc80b4200 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:39.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.456+0000 7f5fc759e640 10 monclient(hunting): picked mon.noname-b con 0x7f5fc80b4920 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:39.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.456+0000 7f5fc759e640 10 monclient(hunting): picked mon.noname-a con 0x7f5fc813f9a0 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:39.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.456+0000 7f5fc759e640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc8142d00 con 0x7f5fc813f9a0 2024-03-29T11:44:39.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.456+0000 7f5fc759e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc8141ff0 con 0x7f5fc80b4920 2024-03-29T11:44:39.483 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.456+0000 7f5fc759e640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc8142180 con 0x7f5fc80b4200 2024-03-29T11:44:39.484 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.456+0000 7f5fc759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:39.484 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.456+0000 7f5fc759e640 10 monclient(hunting): authenticate will time out at 1399.651192s 2024-03-29T11:44:39.484 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.459+0000 7f5fc659c640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5fc80b4200 0x7f5fc8142430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:52208/0 (socket says 172.21.15.100:52208) 2024-03-29T11:44:39.484 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.459+0000 7f5fc659c640 1 -- 172.21.15.100:0/1787040819 learned_addr learned my addr 172.21.15.100:0/1787040819 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:39.484 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.459+0000 7f5fbf7fe640 10 client.?.objecter ms_handle_connect 0x7f5fc80b4200 2024-03-29T11:44:39.484 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.459+0000 7f5fbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:39.485 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.460+0000 7f5fbf7fe640 10 client.?.objecter ms_handle_connect 0x7f5fc80b4920 2024-03-29T11:44:39.485 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.460+0000 7f5fbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:39.485 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.460+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221156033 0 0) 0x7f5fc8142180 con 0x7f5fc80b4200 2024-03-29T11:44:39.485 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:39.485 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:39.485 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:39.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:44:39.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fa8004eb0 con 0x7f5fc80b4200 2024-03-29T11:44:39.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185458092 0 0) 0x7f5fc8141ff0 con 0x7f5fc80b4920 2024-03-29T11:44:39.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:39.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:39.486 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:39.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:44:39.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc8142180 con 0x7f5fc80b4920 2024-03-29T11:44:39.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 441433242 0 0) 0x7f5fa8004eb0 con 0x7f5fc80b4200 2024-03-29T11:44:39.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fc8141ff0 con 0x7f5fc80b4200 2024-03-29T11:44:39.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5fb8001980 con 0x7f5fc80b4200 2024-03-29T11:44:39.487 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:39.488 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.461+0000 7f5fbf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:39.488 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 10 monclient(hunting): dump: 2024-03-29T11:44:39.488 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:39.488 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:39.488 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:39.488 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:39.488 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:39.488 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1385249908 0 0) 0x7f5fc8142180 con 0x7f5fc80b4920 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fa8004eb0 con 0x7f5fc80b4920 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2235727681 0 0) 0x7f5fc8141ff0 con 0x7f5fc80b4200 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 >> v1:172.21.15.191:6789/0 conn(0x7f5fc80b4920 legacy=0x7f5fc8137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 >> v1:172.21.15.100:6789/0 conn(0x7f5fc813f9a0 legacy=0x7f5fc81da550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 1 monclient: found mon.c 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:39.489 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:39.490 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc80b2a30 con 0x7f5fc80b4200 2024-03-29T11:44:39.490 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:39.490 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:39.490 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.462+0000 7f5fbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:39.490 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fc759e640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:44:39.490 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fc759e640 10 monclient: _renew_subs 2024-03-29T11:44:39.490 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fc759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:39.490 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fc759e640 1 -- 172.21.15.100:0/1787040819 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc80b2c80 con 0x7f5fc80b4200 2024-03-29T11:44:39.490 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fc759e640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:39.490 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fc759e640 10 monclient: _renew_subs 2024-03-29T11:44:39.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fc759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:39.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fc759e640 1 -- 172.21.15.100:0/1787040819 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc81385c0 con 0x7f5fc80b4200 2024-03-29T11:44:39.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fc759e640 1 librados: init done 2024-03-29T11:44:39.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fb8001ae0 con 0x7f5fc80b4200 2024-03-29T11:44:39.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:39.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.463+0000 7f5fc559a640 4 set_mon_vals no callback set 2024-03-29T11:44:39.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.464+0000 7f5fc759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:39.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.464+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5fb8004a90 con 0x7f5fc80b4200 2024-03-29T11:44:39.491 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.464+0000 7f5fc759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:39.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.464+0000 7f5fc759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:39.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.464+0000 7f5fc759e640 1 -- 172.21.15.100:0/1787040819 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fac005080 con 0x7f5fc80b4200 2024-03-29T11:44:39.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.464+0000 7f5fbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:39.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.464+0000 7f5fbf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:39.492 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.464+0000 7f5fbf7fe640 10 monclient: dump: 2024-03-29T11:44:39.492 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:39.492 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:39.492 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:39.492 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:39.492 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.468+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f5fb8004c70 con 0x7f5fc80b4200 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.469+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f5fb8084a80 con 0x7f5fc80b4200 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.469+0000 7f5fbdd9b640 10 monclient: get_auth_request con 0x7f5fa806ddc0 auth_method 0 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.469+0000 7f5fbf7fe640 10 client.4586.objecter ms_dispatch 0x7f5fc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.469+0000 7f5fbf7fe640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:39.493 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.469+0000 7f5fbf7fe640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:39.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.469+0000 7f5fbf7fe640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:44:39.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.469+0000 7f5fbf7fe640 10 client.4586.objecter ms_handle_connect 0x7f5fa806ddc0 2024-03-29T11:44:39.494 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.477+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fb8050190 con 0x7f5fc80b4200 2024-03-29T11:44:39.495 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.477+0000 7f5fbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:39.495 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.477+0000 7f5fbf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:39.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.798+0000 7f5fc759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:39.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.798+0000 7f5fc759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:39.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.799+0000 7f5fc759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:44:39.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.799+0000 7f5fc759e640 1 -- 172.21.15.100:0/1787040819 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fac005330 con 0x7f5fc80b4200 2024-03-29T11:44:39.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.802+0000 7f5fbf7fe640 1 -- 172.21.15.100:0/1787040819 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f5fb8052290 con 0x7f5fc80b4200 2024-03-29T11:44:39.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.802+0000 7f5fbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:39.807 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.802+0000 7f5fbf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:39.807 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:44:39.807 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:39.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.805+0000 7f5fc759e640 10 librados: watch_flush enter 2024-03-29T11:44:39.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.805+0000 7f5fc759e640 10 librados: watch_flush exit 2024-03-29T11:44:39.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.805+0000 7f5fc759e640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:44:39.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.805+0000 7f5fc759e640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:44:39.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.806+0000 7f5fc759e640 1 -- 172.21.15.100:0/1787040819 >> v1:172.21.15.100:6816/3220289657 conn(0x7f5fa806ddc0 legacy=0x7f5fa8070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:39.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.806+0000 7f5fc759e640 10 monclient: shutdown 2024-03-29T11:44:39.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.806+0000 7f5fc759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:39.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.806+0000 7f5fc759e640 1 -- 172.21.15.100:0/1787040819 >> v1:172.21.15.100:6790/0 conn(0x7f5fc80b4200 legacy=0x7f5fc8142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:39.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.806+0000 7f5fc759e640 1 -- 172.21.15.100:0/1787040819 shutdown_connections 2024-03-29T11:44:39.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.806+0000 7f5fc759e640 1 -- 172.21.15.100:0/1787040819 >> 172.21.15.100:0/1787040819 conn(0x7f5fc806d230 msgr2=0x7f5fc8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:39.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.807+0000 7f5fc759e640 1 -- 172.21.15.100:0/1787040819 shutdown_connections 2024-03-29T11:44:39.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.807+0000 7f5fc759e640 1 -- 172.21.15.100:0/1787040819 wait complete. 2024-03-29T11:44:39.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:39.807+0000 7f5fc759e640 1 librados: shutdown 2024-03-29T11:44:39.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:46.639 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:46.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.782+0000 7f31f1d30640 10 monclient: get_monmap_and_config 2024-03-29T11:44:46.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.782+0000 7f31f1d30640 10 monclient: build_initial_monmap 2024-03-29T11:44:46.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.782+0000 7f31f1d30640 10 monclient: monmap: 2024-03-29T11:44:46.787 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:46.787 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:46.787 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:44:46.783764+0000 2024-03-29T11:44:46.788 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:44:46.783764+0000 2024-03-29T11:44:46.788 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:46.788 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:46.788 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:46.788 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:44:46.788 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:44:46.788 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:46.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.784+0000 7f31f1d30640 1 Processor -- start 2024-03-29T11:44:46.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.784+0000 7f31f1d30640 1 -- start start 2024-03-29T11:44:46.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.784+0000 7f31f1d30640 10 monclient: init 2024-03-29T11:44:46.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31f1d30640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:46.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31f1d30640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:46.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31f1d30640 10 monclient(hunting): picked mon.noname-a con 0x7f31ec06ebd0 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:46.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31f1d30640 10 monclient(hunting): picked mon.noname-c con 0x7f31ec06efb0 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:46.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31f1d30640 10 monclient(hunting): picked mon.noname-b con 0x7f31ec13ed90 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:46.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31f1d30640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec070b40 con 0x7f31ec06ebd0 2024-03-29T11:44:46.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31f1d30640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec070e20 con 0x7f31ec06efb0 2024-03-29T11:44:46.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31f1d30640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec071100 con 0x7f31ec13ed90 2024-03-29T11:44:46.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31f1d30640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:46.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31f1d30640 10 monclient(hunting): authenticate will time out at 1406.980746s 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31eb7fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f31ec06ebd0 0x7f31ec13b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:47520/0 (socket says 172.21.15.100:47520) 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.785+0000 7f31eb7fe640 1 -- 172.21.15.100:0/43569434 learned_addr learned my addr 172.21.15.100:0/43569434 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554146234 0 0) 0x7f31ec070b40 con 0x7f31ec06ebd0 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d4004630 con 0x7f31ec06ebd0 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717888368 0 0) 0x7f31ec071100 con 0x7f31ec13ed90 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:46.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec070b40 con 0x7f31ec13ed90 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412307225 0 0) 0x7f31ec070e20 con 0x7f31ec06efb0 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec071100 con 0x7f31ec06efb0 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1161122971 0 0) 0x7f31d4004630 con 0x7f31ec06ebd0 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 >> v1:172.21.15.100:6790/0 conn(0x7f31ec13ed90 legacy=0x7f31ec149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 >> v1:172.21.15.191:6789/0 conn(0x7f31ec06efb0 legacy=0x7f31ec13e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:46.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:46.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 monclient: found mon.noname-a 2024-03-29T11:44:46.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:46.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:44:46.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec13c040 con 0x7f31ec06ebd0 2024-03-29T11:44:46.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:46.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:46.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:46.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f31e4001690 con 0x7f31ec06ebd0 2024-03-29T11:44:46.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:46.798 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31ea7fc640 10 monclient: dump: 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:46.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31f1d30640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:44:46.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.786+0000 7f31f1d30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:46.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.787+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31e40017f0 con 0x7f31ec06ebd0 2024-03-29T11:44:46.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.787+0000 7f31ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:46.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.787+0000 7f31ea7fc640 1 -- 172.21.15.100:0/43569434 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f31e4001c00 con 0x7f31ec06ebd0 2024-03-29T11:44:46.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.787+0000 7f31ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:46.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.787+0000 7f31ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:46.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.787+0000 7f31ea7fc640 10 monclient: dump: 2024-03-29T11:44:46.800 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:46.800 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:46.800 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.801 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.801 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:46.801 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:46.801 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:46.801 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:46.801 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:46.801 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:46.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.787+0000 7f31f1d30640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:46.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.787+0000 7f31f1d30640 4 set_mon_vals no callback set 2024-03-29T11:44:46.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.788+0000 7f31f1d30640 10 monclient: shutdown 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.788+0000 7f31f1d30640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.788+0000 7f31f1d30640 1 -- 172.21.15.100:0/43569434 >> v1:172.21.15.100:6789/0 conn(0x7f31ec06ebd0 legacy=0x7f31ec13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.788+0000 7f31f1d30640 1 -- 172.21.15.100:0/43569434 shutdown_connections 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.788+0000 7f31f1d30640 1 -- 172.21.15.100:0/43569434 >> 172.21.15.100:0/43569434 conn(0x7f31ec0b4200 msgr2=0x7f31ec0b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.788+0000 7f31f1d30640 1 -- 172.21.15.100:0/43569434 shutdown_connections 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.788+0000 7f31f1d30640 1 -- 172.21.15.100:0/43569434 wait complete. 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.789+0000 7f31f1d30640 10 monclient: build_initial_monmap 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.789+0000 7f31f1d30640 10 monclient: monmap: 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:44:46.802 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.789+0000 7f31f1d30640 1 librados: starting msgr at 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.789+0000 7f31f1d30640 1 librados: starting objecter 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.790+0000 7f31f1d30640 1 Processor -- start 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.790+0000 7f31f1d30640 1 -- start start 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.790+0000 7f31f1d30640 1 librados: setting wanted keys 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.790+0000 7f31f1d30640 1 librados: calling monclient init 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.790+0000 7f31f1d30640 10 monclient: init 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31f1d30640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31f1d30640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:46.803 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31f1d30640 10 monclient(hunting): picked mon.noname-c con 0x7f31ec06ebd0 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:46.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31f1d30640 10 monclient(hunting): picked mon.noname-b con 0x7f31ec06efb0 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:46.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31f1d30640 10 monclient(hunting): picked mon.noname-a con 0x7f31ec13ed90 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:46.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31f1d30640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec1ccc60 con 0x7f31ec13ed90 2024-03-29T11:44:46.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31f1d30640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec1cb780 con 0x7f31ec06efb0 2024-03-29T11:44:46.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31f1d30640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec1c8210 con 0x7f31ec06ebd0 2024-03-29T11:44:46.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31f1d30640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:46.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31f1d30640 10 monclient(hunting): authenticate will time out at 1406.986441s 2024-03-29T11:44:46.808 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31ebfff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f31ec13ed90 0x7f31ec1e60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:47950/0 (socket says 172.21.15.100:47950) 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31ebfff640 1 -- 172.21.15.100:0/3134939403 learned_addr learned my addr 172.21.15.100:0/3134939403 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31d3fff640 10 client.?.objecter ms_handle_connect 0x7f31ec13ed90 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.791+0000 7f31d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 client.?.objecter ms_handle_connect 0x7f31ec06efb0 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 client.?.objecter ms_handle_connect 0x7f31ec06ebd0 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824610829 0 0) 0x7f31ec1ccc60 con 0x7f31ec13ed90 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d8004eb0 con 0x7f31ec13ed90 2024-03-29T11:44:46.809 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272833081 0 0) 0x7f31ec1cb780 con 0x7f31ec06efb0 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec1ccc60 con 0x7f31ec06efb0 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874542334 0 0) 0x7f31ec1c8210 con 0x7f31ec06ebd0 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec1cb780 con 0x7f31ec06ebd0 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 773999356 0 0) 0x7f31d8004eb0 con 0x7f31ec13ed90 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31ec1c8210 con 0x7f31ec13ed90 2024-03-29T11:44:46.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f31e0001910 con 0x7f31ec13ed90 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): dump: 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:46.811 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3768422216 0 0) 0x7f31ec1ccc60 con 0x7f31ec06efb0 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31d8004eb0 con 0x7f31ec06efb0 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f31dc001bf0 con 0x7f31ec06efb0 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 10 monclient(hunting): dump: 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.812 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1441661323 0 0) 0x7f31ec1cb780 con 0x7f31ec06ebd0 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.792+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31ec1ccc60 con 0x7f31ec06ebd0 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f31e4001b00 con 0x7f31ec06ebd0 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 10 monclient(hunting): dump: 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:46.813 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 778088201 0 0) 0x7f31ec1c8210 con 0x7f31ec13ed90 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 >> v1:172.21.15.100:6790/0 conn(0x7f31ec06ebd0 legacy=0x7f31ec1cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 >> v1:172.21.15.191:6789/0 conn(0x7f31ec06efb0 legacy=0x7f31ec0b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 1 monclient: found mon.a 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:46.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec1c84f0 con 0x7f31ec13ed90 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31f1d30640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31f1d30640 10 monclient: _renew_subs 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31f1d30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31f1d30640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ec1ca500 con 0x7f31ec13ed90 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31f1d30640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31f1d30640 10 monclient: _renew_subs 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31f1d30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31f1d30640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ec0b2df0 con 0x7f31ec13ed90 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.793+0000 7f31f1d30640 1 librados: init done 2024-03-29T11:44:46.815 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.794+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31e00035f0 con 0x7f31ec13ed90 2024-03-29T11:44:46.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.794+0000 7f31d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:46.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.794+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f31e0003940 con 0x7f31ec13ed90 2024-03-29T11:44:46.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.794+0000 7f31d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:46.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.794+0000 7f31d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:46.816 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.794+0000 7f31d3fff640 10 monclient: dump: 2024-03-29T11:44:46.816 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:46.816 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:46.816 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.816 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.794+0000 7f31e9ffb640 4 set_mon_vals no callback set 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.795+0000 7f31f1d30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.796+0000 7f31f1d30640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.796+0000 7f31f1d30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.796+0000 7f31f1d30640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31cc005080 con 0x7f31ec13ed90 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.796+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f31e0003aa0 con 0x7f31ec13ed90 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.797+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f31e0084a30 con 0x7f31ec13ed90 2024-03-29T11:44:46.817 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.797+0000 7f31d3fff640 10 client.4494.objecter ms_dispatch 0x7f31ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:46.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.797+0000 7f31d3fff640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:46.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.797+0000 7f31d3fff640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:46.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.797+0000 7f31d3fff640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-29T11:44:46.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.797+0000 7f31eb7fe640 10 monclient: get_auth_request con 0x7f31d806e330 auth_method 0 2024-03-29T11:44:46.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.798+0000 7f31d3fff640 10 client.4494.objecter ms_handle_connect 0x7f31d806e330 2024-03-29T11:44:46.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.804+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31e0050310 con 0x7f31ec13ed90 2024-03-29T11:44:46.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.805+0000 7f31d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:46.818 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:46.805+0000 7f31d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:47.179 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.175+0000 7f31f1d30640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:47.179 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.175+0000 7f31f1d30640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:47.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.175+0000 7f31f1d30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:47.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.175+0000 7f31f1d30640 1 -- 172.21.15.100:0/3134939403 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31cc005330 con 0x7f31ec13ed90 2024-03-29T11:44:47.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.176+0000 7f31d3fff640 1 -- 172.21.15.100:0/3134939403 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f31e0052240 con 0x7f31ec13ed90 2024-03-29T11:44:47.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.176+0000 7f31d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:47.180 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.176+0000 7f31d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:47.180 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:44:47.181 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:47.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.179+0000 7f31f1d30640 10 librados: watch_flush enter 2024-03-29T11:44:47.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.179+0000 7f31f1d30640 10 librados: watch_flush exit 2024-03-29T11:44:47.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.179+0000 7f31f1d30640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-29T11:44:47.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.179+0000 7f31f1d30640 10 client.4494.objecter successfully canceled tick 2024-03-29T11:44:47.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.180+0000 7f31f1d30640 1 -- 172.21.15.100:0/3134939403 >> v1:172.21.15.100:6816/3220289657 conn(0x7f31d806e330 legacy=0x7f31d80707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:47.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.180+0000 7f31f1d30640 10 monclient: shutdown 2024-03-29T11:44:47.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.180+0000 7f31f1d30640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:47.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.180+0000 7f31f1d30640 1 -- 172.21.15.100:0/3134939403 >> v1:172.21.15.100:6789/0 conn(0x7f31ec13ed90 legacy=0x7f31ec1e60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:47.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.180+0000 7f31f1d30640 1 -- 172.21.15.100:0/3134939403 shutdown_connections 2024-03-29T11:44:47.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.180+0000 7f31f1d30640 1 -- 172.21.15.100:0/3134939403 >> 172.21.15.100:0/3134939403 conn(0x7f31ec0b4200 msgr2=0x7f31ec13b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:47.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.180+0000 7f31f1d30640 1 -- 172.21.15.100:0/3134939403 shutdown_connections 2024-03-29T11:44:47.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.181+0000 7f31f1d30640 1 -- 172.21.15.100:0/3134939403 wait complete. 2024-03-29T11:44:47.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:47.181+0000 7f31f1d30640 1 librados: shutdown 2024-03-29T11:44:47.198 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:54.006 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:54.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.148+0000 7ff0bb672640 10 monclient: get_monmap_and_config 2024-03-29T11:44:54.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.148+0000 7ff0bb672640 10 monclient: build_initial_monmap 2024-03-29T11:44:54.152 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.148+0000 7ff0bb672640 10 monclient: monmap: 2024-03-29T11:44:54.152 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:54.152 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:54.152 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:44:54.149078+0000 2024-03-29T11:44:54.152 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:44:54.149078+0000 2024-03-29T11:44:54.153 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:54.153 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:54.153 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:54.153 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:44:54.153 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:44:54.153 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:54.153 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.149+0000 7ff0bb672640 1 Processor -- start 2024-03-29T11:44:54.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.149+0000 7ff0bb672640 1 -- start start 2024-03-29T11:44:54.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.149+0000 7ff0bb672640 10 monclient: init 2024-03-29T11:44:54.154 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.150+0000 7ff0bb672640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:54.155 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.150+0000 7ff0bb672640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:54.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.150+0000 7ff0bb672640 10 monclient(hunting): picked mon.noname-a con 0x7ff0b41457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:54.158 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.154+0000 7ff0b940e640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff0b41457f0 0x7ff0b413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:51655/0 (socket says 172.21.15.100:51655) 2024-03-29T11:44:54.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.157+0000 7ff0bb672640 10 monclient(hunting): picked mon.noname-c con 0x7ff0b413f650 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:54.166 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.158+0000 7ff0b940e640 1 -- 172.21.15.100:0/4257006140 learned_addr learned my addr 172.21.15.100:0/4257006140 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:54.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.158+0000 7ff0bb672640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b40a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:54.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.158+0000 7ff0bb672640 1 -- 172.21.15.100:0/4257006140 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b40aaef0 con 0x7ff0b41457f0 2024-03-29T11:44:54.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.158+0000 7ff0bb672640 1 -- 172.21.15.100:0/4257006140 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b4145bd0 con 0x7ff0b413f650 2024-03-29T11:44:54.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.158+0000 7ff0bb672640 1 -- 172.21.15.100:0/4257006140 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b4145eb0 con 0x7ff0b40a9d70 2024-03-29T11:44:54.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.158+0000 7ff0bb672640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:54.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.158+0000 7ff0bb672640 10 monclient(hunting): authenticate will time out at 1414.353966s 2024-03-29T11:44:54.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739455268 0 0) 0x7ff0b4145eb0 con 0x7ff0b40a9d70 2024-03-29T11:44:54.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:54.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:54.167 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:54.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:44:54.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a0003520 con 0x7ff0b40a9d70 2024-03-29T11:44:54.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290815924 0 0) 0x7ff0b40aaef0 con 0x7ff0b41457f0 2024-03-29T11:44:54.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:54.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:54.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:54.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:44:54.168 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b4145eb0 con 0x7ff0b41457f0 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2677840095 0 0) 0x7ff0a0003520 con 0x7ff0b40a9d70 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 >> v1:172.21.15.191:6789/0 conn(0x7ff0b413f650 legacy=0x7ff0b40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 >> v1:172.21.15.100:6789/0 conn(0x7ff0b41457f0 legacy=0x7ff0b413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 1 monclient: found mon.noname-b 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b4146340 con 0x7ff0b40a9d70 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.159+0000 7ff0abfff640 10 monclient: _finish_auth 0 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:54.169 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:54.170 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff0ac001a10 con 0x7ff0b40a9d70 2024-03-29T11:44:54.170 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0bb672640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:44:54.170 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0bb672640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:54.170 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:54.170 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 10 monclient: dump: 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0ac001c30 con 0x7ff0b40a9d70 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 1 -- 172.21.15.100:0/4257006140 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff0ac001fc0 con 0x7ff0b40a9d70 2024-03-29T11:44:54.171 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0abfff640 10 monclient: dump: 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0bb672640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:54.172 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.160+0000 7ff0bb672640 4 set_mon_vals no callback set 2024-03-29T11:44:54.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.161+0000 7ff0bb672640 10 monclient: shutdown 2024-03-29T11:44:54.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.161+0000 7ff0bb672640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:54.173 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.161+0000 7ff0bb672640 1 -- 172.21.15.100:0/4257006140 >> v1:172.21.15.100:6790/0 conn(0x7ff0b40a9d70 legacy=0x7ff0b414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.163+0000 7ff0bb672640 1 -- 172.21.15.100:0/4257006140 shutdown_connections 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.163+0000 7ff0bb672640 1 -- 172.21.15.100:0/4257006140 >> 172.21.15.100:0/4257006140 conn(0x7ff0b4066a00 msgr2=0x7ff0b4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.163+0000 7ff0bb672640 1 -- 172.21.15.100:0/4257006140 shutdown_connections 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.164+0000 7ff0bb672640 1 -- 172.21.15.100:0/4257006140 wait complete. 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.165+0000 7ff0bb672640 10 monclient: build_initial_monmap 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.165+0000 7ff0bb672640 10 monclient: monmap: 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:54.181 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.165+0000 7ff0bb672640 1 librados: starting msgr at 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.165+0000 7ff0bb672640 1 librados: starting objecter 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.165+0000 7ff0bb672640 1 Processor -- start 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.166+0000 7ff0bb672640 1 -- start start 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.166+0000 7ff0bb672640 1 librados: setting wanted keys 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.166+0000 7ff0bb672640 1 librados: calling monclient init 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.166+0000 7ff0bb672640 10 monclient: init 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.166+0000 7ff0bb672640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.166+0000 7ff0bb672640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:54.182 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.167+0000 7ff0bb672640 10 monclient(hunting): picked mon.noname-a con 0x7ff0b40a9d70 addr v1:172.21.15.100:6789/0 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.167+0000 7ff0bb672640 10 monclient(hunting): picked mon.noname-c con 0x7ff0b413f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.167+0000 7ff0b8c0d640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7ff0b413f650 0x7ff0b4133a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:44536/0 (socket says 172.21.15.100:44536) 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.168+0000 7ff0b8c0d640 1 -- 172.21.15.100:0/1271963550 learned_addr learned my addr 172.21.15.100:0/1271963550 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.168+0000 7ff0bb672640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b41457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.168+0000 7ff0bb672640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b41427b0 con 0x7ff0b40a9d70 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.168+0000 7ff0bb672640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b4134880 con 0x7ff0b41457f0 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.168+0000 7ff0bb672640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b4135bd0 con 0x7ff0b413f650 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.168+0000 7ff0bb672640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.168+0000 7ff0bb672640 10 monclient(hunting): authenticate will time out at 1414.363321s 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.170+0000 7ff0a9ffb640 10 client.?.objecter ms_handle_connect 0x7ff0b413f650 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.170+0000 7ff0a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 client.?.objecter ms_handle_connect 0x7ff0b40a9d70 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:54.183 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 client.?.objecter ms_handle_connect 0x7ff0b41457f0 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963205001 0 0) 0x7ff0b41427b0 con 0x7ff0b40a9d70 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff098004eb0 con 0x7ff0b40a9d70 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256862858 0 0) 0x7ff0b4135bd0 con 0x7ff0b413f650 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b41427b0 con 0x7ff0b413f650 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2916577429 0 0) 0x7ff098004eb0 con 0x7ff0b40a9d70 2024-03-29T11:44:54.184 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0b4135bd0 con 0x7ff0b40a9d70 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff0a4001780 con 0x7ff0b40a9d70 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.171+0000 7ff0a9ffb640 10 monclient(hunting): dump: 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:54.185 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487127723 0 0) 0x7ff0b4134880 con 0x7ff0b41457f0 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff098004eb0 con 0x7ff0b41457f0 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1746654895 0 0) 0x7ff0b4135bd0 con 0x7ff0b40a9d70 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 >> v1:172.21.15.100:6790/0 conn(0x7ff0b413f650 legacy=0x7ff0b4133a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 >> v1:172.21.15.191:6789/0 conn(0x7ff0b41457f0 legacy=0x7ff0b4134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 1 monclient: found mon.a 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:54.186 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b4135eb0 con 0x7ff0b40a9d70 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0bb672640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0bb672640 10 monclient: _renew_subs 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0bb672640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0bb672640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b4142a00 con 0x7ff0b40a9d70 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0bb672640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0bb672640 10 monclient: _renew_subs 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0bb672640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0bb672640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b4142390 con 0x7ff0b40a9d70 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.172+0000 7ff0bb672640 1 librados: init done 2024-03-29T11:44:54.187 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.173+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0a4001c90 con 0x7ff0b40a9d70 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.173+0000 7ff0a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.173+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff0a4004980 con 0x7ff0b40a9d70 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.173+0000 7ff0a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.173+0000 7ff0a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.173+0000 7ff0a9ffb640 10 monclient: dump: 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:44:54.188 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:44:54.189 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:44:54.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.175+0000 7ff09f7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:54.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.175+0000 7ff09f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:54.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.175+0000 7ff09f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:54.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.175+0000 7ff09f7fe640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b4075f10 con 0x7ff0b40a9d70 2024-03-29T11:44:54.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.175+0000 7ff0aaffd640 4 set_mon_vals no callback set 2024-03-29T11:44:54.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.175+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7ff0a401a950 con 0x7ff0b40a9d70 2024-03-29T11:44:54.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.176+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7ff0a4084840 con 0x7ff0b40a9d70 2024-03-29T11:44:54.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.176+0000 7ff0a9ffb640 10 client.4500.objecter ms_dispatch 0x7ff0b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:54.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.176+0000 7ff0a9ffb640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:54.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.176+0000 7ff0a9ffb640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:54.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.176+0000 7ff0a9ffb640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T11:44:54.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.183+0000 7ff0b8c0d640 10 monclient: get_auth_request con 0x7ff09806de60 auth_method 0 2024-03-29T11:44:54.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.184+0000 7ff0a9ffb640 10 client.4500.objecter ms_handle_connect 0x7ff09806de60 2024-03-29T11:44:54.191 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.187+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0a4050120 con 0x7ff0b40a9d70 2024-03-29T11:44:54.192 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.187+0000 7ff0a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:54.192 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.187+0000 7ff0a9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:54.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.524+0000 7ff09f7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:54.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.524+0000 7ff09f7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:54.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.524+0000 7ff09f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:44:54.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.524+0000 7ff09f7fe640 1 -- 172.21.15.100:0/1271963550 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0b40b3680 con 0x7ff0b40a9d70 2024-03-29T11:44:54.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.525+0000 7ff0a9ffb640 1 -- 172.21.15.100:0/1271963550 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7ff0a4052050 con 0x7ff0b40a9d70 2024-03-29T11:44:54.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.525+0000 7ff0a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:54.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.525+0000 7ff0a9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:54.534 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:44:54.534 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:54.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.530+0000 7ff09f7fe640 10 librados: watch_flush enter 2024-03-29T11:44:54.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 10 librados: watch_flush exit 2024-03-29T11:44:54.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T11:44:54.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 10 client.4500.objecter successfully canceled tick 2024-03-29T11:44:54.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 1 -- 172.21.15.100:0/1271963550 >> v1:172.21.15.100:6816/3220289657 conn(0x7ff09806de60 legacy=0x7ff098070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:54.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 10 monclient: shutdown 2024-03-29T11:44:54.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:54.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 1 -- 172.21.15.100:0/1271963550 >> v1:172.21.15.100:6789/0 conn(0x7ff0b40a9d70 legacy=0x7ff0b4141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:54.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 1 -- 172.21.15.100:0/1271963550 shutdown_connections 2024-03-29T11:44:54.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 1 -- 172.21.15.100:0/1271963550 >> 172.21.15.100:0/1271963550 conn(0x7ff0b4066a00 msgr2=0x7ff0b413ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:54.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 1 -- 172.21.15.100:0/1271963550 shutdown_connections 2024-03-29T11:44:54.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 1 -- 172.21.15.100:0/1271963550 wait complete. 2024-03-29T11:44:54.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:44:54.531+0000 7ff09f7fe640 1 librados: shutdown 2024-03-29T11:44:54.552 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:01.360 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:01.524 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.519+0000 7f02e7af6640 10 monclient: get_monmap_and_config 2024-03-29T11:45:01.524 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.519+0000 7f02e7af6640 10 monclient: build_initial_monmap 2024-03-29T11:45:01.524 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.519+0000 7f02e7af6640 10 monclient: monmap: 2024-03-29T11:45:01.525 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:01.525 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:01.525 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:45:01.520642+0000 2024-03-29T11:45:01.525 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:45:01.520642+0000 2024-03-29T11:45:01.525 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:01.525 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:01.525 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:01.525 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:45:01.525 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:45:01.525 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:01.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.521+0000 7f02e7af6640 1 Processor -- start 2024-03-29T11:45:01.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.521+0000 7f02e7af6640 1 -- start start 2024-03-29T11:45:01.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.521+0000 7f02e7af6640 10 monclient: init 2024-03-29T11:45:01.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e7af6640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:01.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e7af6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:01.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e7af6640 10 monclient(hunting): picked mon.noname-a con 0x7f02e00a9ba0 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:01.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e7af6640 10 monclient(hunting): picked mon.noname-b con 0x7f02e013b5a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:01.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e7af6640 10 monclient(hunting): picked mon.noname-c con 0x7f02e00ab550 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e7af6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e00ac430 con 0x7f02e00a9ba0 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e7af6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e00ac680 con 0x7f02e00ab550 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e7af6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e00ac960 con 0x7f02e013b5a0 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e7af6640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e5091640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f02e013b5a0 0x7f02e00aae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:42676/0 (socket says 172.21.15.100:42676) 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e5091640 1 -- 172.21.15.100:0/3431991282 learned_addr learned my addr 172.21.15.100:0/3431991282 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.522+0000 7f02e7af6640 10 monclient(hunting): authenticate will time out at 1421.717985s 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.523+0000 7f02e4890640 1 -- 172.21.15.100:0/3431991282 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484215775 0 0) 0x7f02e00ac960 con 0x7f02e013b5a0 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.523+0000 7f02e4890640 10 monclient(hunting): _negotiate 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.523+0000 7f02e4890640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.523+0000 7f02e4890640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:01.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.523+0000 7f02e4890640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:45:01.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.523+0000 7f02e4890640 1 -- 172.21.15.100:0/3431991282 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02cc003520 con 0x7f02e013b5a0 2024-03-29T11:45:01.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 1 -- 172.21.15.100:0/3431991282 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1981173347 0 0) 0x7f02cc003520 con 0x7f02e013b5a0 2024-03-29T11:45:01.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 1 -- 172.21.15.100:0/3431991282 >> v1:172.21.15.191:6789/0 conn(0x7f02e00ab550 legacy=0x7f02e01498d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:01.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 1 -- 172.21.15.100:0/3431991282 >> v1:172.21.15.100:6789/0 conn(0x7f02e00a9ba0 legacy=0x7f02e013ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:01.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 10 monclient: _finish_hunting 0 2024-03-29T11:45:01.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 1 monclient: found mon.noname-b 2024-03-29T11:45:01.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:01.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:45:01.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 1 -- 172.21.15.100:0/3431991282 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e013baf0 con 0x7f02e013b5a0 2024-03-29T11:45:01.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 10 monclient: _finish_auth 0 2024-03-29T11:45:01.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 10 monclient: _check_auth_tickets 2024-03-29T11:45:01.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:01.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 1 -- 172.21.15.100:0/3431991282 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f02d0001690 con 0x7f02e013b5a0 2024-03-29T11:45:01.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:01.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:01.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e4890640 10 monclient: dump: 2024-03-29T11:45:01.533 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:01.533 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:01.533 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:01.533 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:01.533 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:01.533 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:01.533 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:01.533 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:01.533 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:01.533 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:01.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e7af6640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:45:01.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.526+0000 7f02e7af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:01.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.528+0000 7f02e4890640 1 -- 172.21.15.100:0/3431991282 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02d0001860 con 0x7f02e013b5a0 2024-03-29T11:45:01.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.528+0000 7f02e4890640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:01.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.528+0000 7f02e4890640 1 -- 172.21.15.100:0/3431991282 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f02d0001cb0 con 0x7f02e013b5a0 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.528+0000 7f02e4890640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.528+0000 7f02e4890640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.528+0000 7f02e4890640 10 monclient: dump: 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.529+0000 7f02e7af6640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:01.536 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.529+0000 7f02e7af6640 4 set_mon_vals no callback set 2024-03-29T11:45:01.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.530+0000 7f02e7af6640 10 monclient: shutdown 2024-03-29T11:45:01.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.530+0000 7f02e7af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:01.537 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.530+0000 7f02e7af6640 1 -- 172.21.15.100:0/3431991282 >> v1:172.21.15.100:6790/0 conn(0x7f02e013b5a0 legacy=0x7f02e00aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:01.539 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.532+0000 7f02e7af6640 1 -- 172.21.15.100:0/3431991282 shutdown_connections 2024-03-29T11:45:01.539 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.532+0000 7f02e7af6640 1 -- 172.21.15.100:0/3431991282 >> 172.21.15.100:0/3431991282 conn(0x7f02e00b3a00 msgr2=0x7f02e00b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:01.539 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.532+0000 7f02e7af6640 1 -- 172.21.15.100:0/3431991282 shutdown_connections 2024-03-29T11:45:01.540 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.532+0000 7f02e7af6640 1 -- 172.21.15.100:0/3431991282 wait complete. 2024-03-29T11:45:01.540 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.533+0000 7f02e7af6640 10 monclient: build_initial_monmap 2024-03-29T11:45:01.540 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.533+0000 7f02e7af6640 10 monclient: monmap: 2024-03-29T11:45:01.540 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:01.540 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:01.540 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:45:01.540 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:45:01.541 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:01.541 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:01.541 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:01.541 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:45:01.541 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:45:01.541 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:01.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.533+0000 7f02e7af6640 1 librados: starting msgr at 2024-03-29T11:45:01.541 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.533+0000 7f02e7af6640 1 librados: starting objecter 2024-03-29T11:45:01.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.533+0000 7f02e7af6640 1 Processor -- start 2024-03-29T11:45:01.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 1 -- start start 2024-03-29T11:45:01.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 1 librados: setting wanted keys 2024-03-29T11:45:01.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 1 librados: calling monclient init 2024-03-29T11:45:01.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 10 monclient: init 2024-03-29T11:45:01.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:01.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:01.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 10 monclient(hunting): picked mon.noname-b con 0x7f02e00a9ba0 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:01.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 10 monclient(hunting): picked mon.noname-a con 0x7f02e00ab550 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:01.542 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 10 monclient(hunting): picked mon.noname-c con 0x7f02e013b5a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:01.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e01417f0 con 0x7f02e00ab550 2024-03-29T11:45:01.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e01406f0 con 0x7f02e00a9ba0 2024-03-29T11:45:01.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e0140880 con 0x7f02e013b5a0 2024-03-29T11:45:01.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:01.543 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.534+0000 7f02e7af6640 10 monclient(hunting): authenticate will time out at 1421.730056s 2024-03-29T11:45:01.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.537+0000 7f02e6093640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f02e013b5a0 0x7f02e01de6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:45740/0 (socket says 172.21.15.100:45740) 2024-03-29T11:45:01.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.537+0000 7f02e6093640 1 -- 172.21.15.100:0/251079250 learned_addr learned my addr 172.21.15.100:0/251079250 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:01.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.538+0000 7f02d67fc640 10 client.?.objecter ms_handle_connect 0x7f02e00ab550 2024-03-29T11:45:01.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.538+0000 7f02d67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:01.544 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.539+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161376487 0 0) 0x7f02e01417f0 con 0x7f02e00ab550 2024-03-29T11:45:01.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.539+0000 7f02d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:01.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.539+0000 7f02d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:01.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.539+0000 7f02d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:01.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.539+0000 7f02d67fc640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:45:01.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.539+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c4004eb0 con 0x7f02e00ab550 2024-03-29T11:45:01.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.539+0000 7f02d67fc640 10 client.?.objecter ms_handle_connect 0x7f02e013b5a0 2024-03-29T11:45:01.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.539+0000 7f02d67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:01.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.540+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627218991 0 0) 0x7f02e0140880 con 0x7f02e013b5a0 2024-03-29T11:45:01.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.540+0000 7f02d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:01.545 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.540+0000 7f02d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:01.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.540+0000 7f02d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:01.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.540+0000 7f02d67fc640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:45:01.546 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.540+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e01417f0 con 0x7f02e013b5a0 2024-03-29T11:45:01.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3170052991 0 0) 0x7f02c4004eb0 con 0x7f02e00ab550 2024-03-29T11:45:01.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02e0140880 con 0x7f02e00ab550 2024-03-29T11:45:01.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f02d0001b70 con 0x7f02e00ab550 2024-03-29T11:45:01.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:01.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:01.547 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 10 monclient(hunting): dump: 2024-03-29T11:45:01.547 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 10 client.?.objecter ms_handle_connect 0x7f02e00a9ba0 2024-03-29T11:45:01.548 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:01.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3703005864 0 0) 0x7f02e0140880 con 0x7f02e00ab550 2024-03-29T11:45:01.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 >> v1:172.21.15.100:6790/0 conn(0x7f02e013b5a0 legacy=0x7f02e01de6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:01.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 >> v1:172.21.15.191:6789/0 conn(0x7f02e00a9ba0 legacy=0x7f02e01359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:01.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:01.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 1 monclient: found mon.a 2024-03-29T11:45:01.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:01.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:01.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e013dd40 con 0x7f02e00ab550 2024-03-29T11:45:01.549 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02e7af6640 5 monclient: authenticate success, global_id 4506 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02e7af6640 10 monclient: _renew_subs 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02e7af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02e7af6640 1 -- 172.21.15.100:0/251079250 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e013df90 con 0x7f02e00ab550 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02e7af6640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02e7af6640 10 monclient: _renew_subs 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02e7af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.541+0000 7f02e7af6640 1 -- 172.21.15.100:0/251079250 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e01362e0 con 0x7f02e00ab550 2024-03-29T11:45:01.550 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.542+0000 7f02e7af6640 1 librados: init done 2024-03-29T11:45:01.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.543+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02d0004a90 con 0x7f02e00ab550 2024-03-29T11:45:01.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.543+0000 7f02d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:01.551 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.543+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f02d0001250 con 0x7f02e00ab550 2024-03-29T11:45:01.552 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.543+0000 7f02d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:01.552 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.543+0000 7f02d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:01.552 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.543+0000 7f02d67fc640 10 monclient: dump: 2024-03-29T11:45:01.552 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:01.552 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:01.552 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:01.552 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:01.552 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:01.552 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:01.553 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:01.553 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:01.553 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:01.553 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:01.553 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.543+0000 7f02d7fff640 4 set_mon_vals no callback set 2024-03-29T11:45:01.553 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.543+0000 7f02e7af6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:01.553 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.544+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f02d001ac10 con 0x7f02e00ab550 2024-03-29T11:45:01.553 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.544+0000 7f02e7af6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:01.553 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.544+0000 7f02e7af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:01.553 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.545+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f02d0084ad0 con 0x7f02e00ab550 2024-03-29T11:45:01.554 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.545+0000 7f02d67fc640 10 client.4506.objecter ms_dispatch 0x7f02e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:01.554 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.545+0000 7f02d67fc640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:01.554 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.545+0000 7f02d67fc640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:01.554 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.545+0000 7f02d67fc640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-29T11:45:01.554 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.544+0000 7f02e7af6640 1 -- 172.21.15.100:0/251079250 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c8005080 con 0x7f02e00ab550 2024-03-29T11:45:01.554 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.545+0000 7f02e5892640 10 monclient: get_auth_request con 0x7f02c406ddd0 auth_method 0 2024-03-29T11:45:01.554 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.545+0000 7f02d67fc640 10 client.4506.objecter ms_handle_connect 0x7f02c406ddd0 2024-03-29T11:45:01.561 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.556+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02d00503b0 con 0x7f02e00ab550 2024-03-29T11:45:01.561 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.556+0000 7f02d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:01.561 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.556+0000 7f02d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:01.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.898+0000 7f02e7af6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:01.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.898+0000 7f02e7af6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:01.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.898+0000 7f02e7af6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:01.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.898+0000 7f02e7af6640 1 -- 172.21.15.100:0/251079250 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02c8005330 con 0x7f02e00ab550 2024-03-29T11:45:01.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.900+0000 7f02d67fc640 1 -- 172.21.15.100:0/251079250 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f02d00522e0 con 0x7f02e00ab550 2024-03-29T11:45:01.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.900+0000 7f02d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:01.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.900+0000 7f02d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:01.905 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:45:01.905 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:01.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.904+0000 7f02e7af6640 10 librados: watch_flush enter 2024-03-29T11:45:01.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.904+0000 7f02e7af6640 10 librados: watch_flush exit 2024-03-29T11:45:01.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.904+0000 7f02e7af6640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-29T11:45:01.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.904+0000 7f02e7af6640 10 client.4506.objecter successfully canceled tick 2024-03-29T11:45:01.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.904+0000 7f02e7af6640 1 -- 172.21.15.100:0/251079250 >> v1:172.21.15.100:6816/3220289657 conn(0x7f02c406ddd0 legacy=0x7f02c4070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:01.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.904+0000 7f02e7af6640 10 monclient: shutdown 2024-03-29T11:45:01.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.904+0000 7f02e7af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:01.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.904+0000 7f02e7af6640 1 -- 172.21.15.100:0/251079250 >> v1:172.21.15.100:6789/0 conn(0x7f02e00ab550 legacy=0x7f02e01d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:01.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.905+0000 7f02e7af6640 1 -- 172.21.15.100:0/251079250 shutdown_connections 2024-03-29T11:45:01.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.905+0000 7f02e7af6640 1 -- 172.21.15.100:0/251079250 >> 172.21.15.100:0/251079250 conn(0x7f02e00b3a00 msgr2=0x7f02e0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:01.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.906+0000 7f02e7af6640 1 -- 172.21.15.100:0/251079250 shutdown_connections 2024-03-29T11:45:01.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.907+0000 7f02e7af6640 1 -- 172.21.15.100:0/251079250 wait complete. 2024-03-29T11:45:01.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:01.907+0000 7f02e7af6640 1 librados: shutdown 2024-03-29T11:45:01.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:08.735 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:08.886 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.881+0000 7f5bd890d640 10 monclient: get_monmap_and_config 2024-03-29T11:45:08.886 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.881+0000 7f5bd890d640 10 monclient: build_initial_monmap 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.881+0000 7f5bd890d640 10 monclient: monmap: 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:45:08.882925+0000 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:45:08.882925+0000 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:45:08.887 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:08.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.883+0000 7f5bd890d640 1 Processor -- start 2024-03-29T11:45:08.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.883+0000 7f5bd890d640 1 -- start start 2024-03-29T11:45:08.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.883+0000 7f5bd890d640 10 monclient: init 2024-03-29T11:45:08.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd890d640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:08.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd890d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:08.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd890d640 10 monclient(hunting): picked mon.noname-a con 0x7f5bcc036a60 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:08.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd890d640 10 monclient(hunting): picked mon.noname-b con 0x7f5bcc0c5de0 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:08.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd890d640 10 monclient(hunting): picked mon.noname-c con 0x7f5bcc0c3070 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:08.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd890d640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc02cc60 con 0x7f5bcc036a60 2024-03-29T11:45:08.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd890d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc02e040 con 0x7f5bcc0c3070 2024-03-29T11:45:08.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd890d640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc02e320 con 0x7f5bcc0c5de0 2024-03-29T11:45:08.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd890d640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:08.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd890d640 10 monclient(hunting): authenticate will time out at 1429.079601s 2024-03-29T11:45:08.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd259c640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5bcc0c5de0 0x7f5bcc0c2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:50868/0 (socket says 172.21.15.100:50868) 2024-03-29T11:45:08.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.884+0000 7f5bd259c640 1 -- 172.21.15.100:0/1836186400 learned_addr learned my addr 172.21.15.100:0/1836186400 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:08.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 1 -- 172.21.15.100:0/1836186400 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529609894 0 0) 0x7f5bcc02e320 con 0x7f5bcc0c5de0 2024-03-29T11:45:08.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:45:08.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:08.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:08.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:45:08.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 1 -- 172.21.15.100:0/1836186400 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc003520 con 0x7f5bcc0c5de0 2024-03-29T11:45:08.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 1 -- 172.21.15.100:0/1836186400 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 454197533 0 0) 0x7f5bbc003520 con 0x7f5bcc0c5de0 2024-03-29T11:45:08.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 1 -- 172.21.15.100:0/1836186400 >> v1:172.21.15.191:6789/0 conn(0x7f5bcc0c3070 legacy=0x7f5bcc0d0fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:45:08.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 1 -- 172.21.15.100:0/1836186400 >> v1:172.21.15.100:6789/0 conn(0x7f5bcc036a60 legacy=0x7f5bcc0c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:08.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:45:08.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 1 monclient: found mon.noname-b 2024-03-29T11:45:08.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:08.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:45:08.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 1 -- 172.21.15.100:0/1836186400 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc02c830 con 0x7f5bcc0c5de0 2024-03-29T11:45:08.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:45:08.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:45:08.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:08.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 1 -- 172.21.15.100:0/1836186400 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5bc8001690 con 0x7f5bcc0c5de0 2024-03-29T11:45:08.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd890d640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:45:08.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd890d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:08.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:08.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:08.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd1d9b640 10 monclient: dump: 2024-03-29T11:45:08.894 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:08.894 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:08.894 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:08.894 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:08.894 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:08.894 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.885+0000 7f5bd890d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.886+0000 7f5bd1d9b640 1 -- 172.21.15.100:0/1836186400 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bc8001860 con 0x7f5bcc0c5de0 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.886+0000 7f5bd1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.886+0000 7f5bd1d9b640 1 -- 172.21.15.100:0/1836186400 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5bc8001cb0 con 0x7f5bcc0c5de0 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.886+0000 7f5bd1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.886+0000 7f5bd1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.886+0000 7f5bd1d9b640 10 monclient: dump: 2024-03-29T11:45:08.895 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:08.896 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:08.896 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:08.896 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:08.896 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:08.896 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:08.896 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:08.896 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:08.896 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:08.896 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:08.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.886+0000 7f5bd890d640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:08.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.886+0000 7f5bd890d640 4 set_mon_vals no callback set 2024-03-29T11:45:08.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.887+0000 7f5bd890d640 10 monclient: shutdown 2024-03-29T11:45:08.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.887+0000 7f5bd890d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:08.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.887+0000 7f5bd890d640 1 -- 172.21.15.100:0/1836186400 >> v1:172.21.15.100:6790/0 conn(0x7f5bcc0c5de0 legacy=0x7f5bcc0c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:08.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.887+0000 7f5bd890d640 1 -- 172.21.15.100:0/1836186400 shutdown_connections 2024-03-29T11:45:08.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.887+0000 7f5bd890d640 1 -- 172.21.15.100:0/1836186400 >> 172.21.15.100:0/1836186400 conn(0x7f5bcc005c00 msgr2=0x7f5bcc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:08.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.887+0000 7f5bd890d640 1 -- 172.21.15.100:0/1836186400 shutdown_connections 2024-03-29T11:45:08.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.888+0000 7f5bd890d640 1 -- 172.21.15.100:0/1836186400 wait complete. 2024-03-29T11:45:08.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.889+0000 7f5bd890d640 10 monclient: build_initial_monmap 2024-03-29T11:45:08.907 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.889+0000 7f5bd890d640 10 monclient: monmap: 2024-03-29T11:45:08.907 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:08.907 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:08.907 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:45:08.907 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:45:08.907 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:08.907 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.889+0000 7f5bd890d640 1 librados: starting msgr at 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.889+0000 7f5bd890d640 1 librados: starting objecter 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.890+0000 7f5bd890d640 1 Processor -- start 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.890+0000 7f5bd890d640 1 -- start start 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.890+0000 7f5bd890d640 1 librados: setting wanted keys 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.890+0000 7f5bd890d640 1 librados: calling monclient init 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.890+0000 7f5bd890d640 10 monclient: init 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.891+0000 7f5bd890d640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.891+0000 7f5bd890d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.891+0000 7f5bd890d640 10 monclient(hunting): picked mon.noname-c con 0x7f5bcc036a60 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:08.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.891+0000 7f5bd890d640 10 monclient(hunting): picked mon.noname-a con 0x7f5bcc0c3070 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.891+0000 7f5bd2d9d640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f5bcc036a60 0x7f5bcc0343b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:51624/0 (socket says 172.21.15.100:51624) 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.891+0000 7f5bd2d9d640 1 -- 172.21.15.100:0/1366361908 learned_addr learned my addr 172.21.15.100:0/1366361908 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.891+0000 7f5bd890d640 10 monclient(hunting): picked mon.noname-b con 0x7f5bcc0c47e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.891+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc15c640 con 0x7f5bcc0c3070 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.891+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc0c3850 con 0x7f5bcc0c47e0 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.891+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc033af0 con 0x7f5bcc036a60 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bd890d640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bd890d640 10 monclient(hunting): authenticate will time out at 1429.087097s 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 client.?.objecter ms_handle_connect 0x7f5bcc036a60 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 client.?.objecter ms_handle_connect 0x7f5bcc0c3070 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85942758 0 0) 0x7f5bcc033af0 con 0x7f5bcc036a60 2024-03-29T11:45:08.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc0004eb0 con 0x7f5bcc036a60 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85673161 0 0) 0x7f5bcc15c640 con 0x7f5bcc0c3070 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.892+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bcc033af0 con 0x7f5bcc0c3070 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425526725 0 0) 0x7f5bcc033af0 con 0x7f5bcc0c3070 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bcc15c640 con 0x7f5bcc0c3070 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5bc8001b70 con 0x7f5bcc0c3070 2024-03-29T11:45:08.910 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 10 monclient(hunting): dump: 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:08.911 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1955879993 0 0) 0x7f5bc0004eb0 con 0x7f5bcc036a60 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bcc033af0 con 0x7f5bcc036a60 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5bc4001960 con 0x7f5bcc036a60 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 10 monclient(hunting): dump: 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:08.912 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4265291288 0 0) 0x7f5bcc15c640 con 0x7f5bcc0c3070 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 >> v1:172.21.15.100:6790/0 conn(0x7f5bcc036a60 legacy=0x7f5bcc0343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.893+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 >> v1:172.21.15.191:6789/0 conn(0x7f5bcc0c47e0 legacy=0x7f5bcc0c4e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.894+0000 7f5bbb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.894+0000 7f5bbb7fe640 1 monclient: found mon.a 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.894+0000 7f5bbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.894+0000 7f5bbb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.894+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc033df0 con 0x7f5bcc0c3070 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.894+0000 7f5bbb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.894+0000 7f5bbb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.894+0000 7f5bbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bd890d640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bd890d640 10 monclient: _renew_subs 2024-03-29T11:45:08.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bd890d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bcc034ac0 con 0x7f5bcc0c3070 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bd890d640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bd890d640 10 monclient: _renew_subs 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bd890d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bcc15d3e0 con 0x7f5bcc0c3070 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bd890d640 1 librados: init done 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bc8001d50 con 0x7f5bcc0c3070 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bbb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f5bc8004c30 con 0x7f5bcc0c3070 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bbb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.895+0000 7f5bbb7fe640 10 monclient: dump: 2024-03-29T11:45:08.914 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.896+0000 7f5bd890d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.896+0000 7f5bd890d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.896+0000 7f5bd890d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.896+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bcc005120 con 0x7f5bcc0c3070 2024-03-29T11:45:08.915 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.897+0000 7f5bd159a640 4 set_mon_vals no callback set 2024-03-29T11:45:08.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.898+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f5bc8004d90 con 0x7f5bcc0c3070 2024-03-29T11:45:08.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.899+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f5bc8084a20 con 0x7f5bcc0c3070 2024-03-29T11:45:08.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.899+0000 7f5bbb7fe640 10 client.4509.objecter ms_dispatch 0x7f5bd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:08.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.899+0000 7f5bbb7fe640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:08.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.899+0000 7f5bbb7fe640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:08.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.899+0000 7f5bbb7fe640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-29T11:45:08.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.899+0000 7f5bd2d9d640 10 monclient: get_auth_request con 0x7f5bc006de80 auth_method 0 2024-03-29T11:45:08.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.899+0000 7f5bbb7fe640 10 client.4509.objecter ms_handle_connect 0x7f5bc006de80 2024-03-29T11:45:08.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.906+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bc8050300 con 0x7f5bcc0c3070 2024-03-29T11:45:08.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.906+0000 7f5bbb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:08.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:08.906+0000 7f5bbb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:09.256 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.252+0000 7f5bd890d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:09.256 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.252+0000 7f5bd890d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:09.257 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.252+0000 7f5bd890d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:09.257 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.252+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bcc00a900 con 0x7f5bcc0c3070 2024-03-29T11:45:09.258 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.254+0000 7f5bbb7fe640 1 -- 172.21.15.100:0/1366361908 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f5bc8052230 con 0x7f5bcc0c3070 2024-03-29T11:45:09.258 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.254+0000 7f5bbb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:09.258 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.254+0000 7f5bbb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:09.258 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:45:09.259 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:09.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.257+0000 7f5bd890d640 10 librados: watch_flush enter 2024-03-29T11:45:09.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.257+0000 7f5bd890d640 10 librados: watch_flush exit 2024-03-29T11:45:09.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.257+0000 7f5bd890d640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-29T11:45:09.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.257+0000 7f5bd890d640 10 client.4509.objecter successfully canceled tick 2024-03-29T11:45:09.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.257+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 >> v1:172.21.15.100:6816/3220289657 conn(0x7f5bc006de80 legacy=0x7f5bc0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:09.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.257+0000 7f5bd890d640 10 monclient: shutdown 2024-03-29T11:45:09.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.257+0000 7f5bd890d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:09.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.257+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 >> v1:172.21.15.100:6789/0 conn(0x7f5bcc0c3070 legacy=0x7f5bcc15caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:09.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.257+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 shutdown_connections 2024-03-29T11:45:09.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.257+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 >> 172.21.15.100:0/1366361908 conn(0x7f5bcc005c00 msgr2=0x7f5bcc0c2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:09.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.258+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 shutdown_connections 2024-03-29T11:45:09.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.258+0000 7f5bd890d640 1 -- 172.21.15.100:0/1366361908 wait complete. 2024-03-29T11:45:09.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:09.258+0000 7f5bd890d640 1 librados: shutdown 2024-03-29T11:45:09.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:16.085 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:16.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.238+0000 7f81165ca640 10 monclient: get_monmap_and_config 2024-03-29T11:45:16.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.238+0000 7f81165ca640 10 monclient: build_initial_monmap 2024-03-29T11:45:16.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.238+0000 7f81165ca640 10 monclient: monmap: 2024-03-29T11:45:16.243 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:16.243 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:16.243 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:45:16.239453+0000 2024-03-29T11:45:16.243 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:45:16.239453+0000 2024-03-29T11:45:16.243 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:16.243 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:16.243 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:16.244 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:45:16.244 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:45:16.244 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:16.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.239+0000 7f81165ca640 1 Processor -- start 2024-03-29T11:45:16.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.239+0000 7f81165ca640 1 -- start start 2024-03-29T11:45:16.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.239+0000 7f81165ca640 10 monclient: init 2024-03-29T11:45:16.248 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.240+0000 7f81165ca640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.240+0000 7f81165ca640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.240+0000 7f81165ca640 10 monclient(hunting): picked mon.noname-a con 0x7f81101457f0 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.240+0000 7f81165ca640 10 monclient(hunting): picked mon.noname-c con 0x7f811013f650 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.240+0000 7f81165ca640 10 monclient(hunting): picked mon.noname-b con 0x7f81100a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.240+0000 7f81165ca640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81100aaef0 con 0x7f81101457f0 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.240+0000 7f81165ca640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8110145bd0 con 0x7f811013f650 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.240+0000 7f81165ca640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8110145eb0 con 0x7f81100a9d70 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.240+0000 7f81165ca640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.240+0000 7f81165ca640 10 monclient(hunting): authenticate will time out at 1436.435937s 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.241+0000 7f810f7fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f811013f650 0x7f81100b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:57666/0 (socket says 172.21.15.100:57666) 2024-03-29T11:45:16.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.241+0000 7f810f7fe640 1 -- 172.21.15.100:0/2301399431 learned_addr learned my addr 172.21.15.100:0/2301399431 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.241+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483915760 0 0) 0x7f8110145bd0 con 0x7f811013f650 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.241+0000 7f810effd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.241+0000 7f810effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.241+0000 7f810effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.241+0000 7f810effd640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.241+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f8003520 con 0x7f811013f650 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285504452 0 0) 0x7f8110145eb0 con 0x7f81100a9d70 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8110145bd0 con 0x7f81100a9d70 2024-03-29T11:45:16.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139995030 0 0) 0x7f81100aaef0 con 0x7f81101457f0 2024-03-29T11:45:16.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:16.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:16.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:16.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:45:16.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8110145eb0 con 0x7f81101457f0 2024-03-29T11:45:16.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1903255907 0 0) 0x7f8110145eb0 con 0x7f81101457f0 2024-03-29T11:45:16.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 >> v1:172.21.15.100:6790/0 conn(0x7f81100a9d70 legacy=0x7f811014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:16.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 >> v1:172.21.15.191:6789/0 conn(0x7f811013f650 legacy=0x7f81100b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:16.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:16.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 monclient: found mon.noname-a 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8110146340 con 0x7f81101457f0 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: _finish_auth 0 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8100001530 con 0x7f81101457f0 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: dump: 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:16.252 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:16.253 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:16.253 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:16.253 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:16.253 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:16.253 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:16.253 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:16.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f81165ca640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:45:16.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f81165ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:16.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8100001690 con 0x7f81101457f0 2024-03-29T11:45:16.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 1 -- 172.21.15.100:0/2301399431 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8100001a60 con 0x7f81101457f0 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f81165ca640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f81165ca640 4 set_mon_vals no callback set 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.242+0000 7f810effd640 10 monclient: dump: 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:16.254 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:16.255 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:16.255 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:16.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.243+0000 7f81165ca640 10 monclient: shutdown 2024-03-29T11:45:16.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.243+0000 7f81165ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:16.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.243+0000 7f81165ca640 1 -- 172.21.15.100:0/2301399431 >> v1:172.21.15.100:6789/0 conn(0x7f81101457f0 legacy=0x7f811013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:16.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.243+0000 7f81165ca640 1 -- 172.21.15.100:0/2301399431 shutdown_connections 2024-03-29T11:45:16.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.243+0000 7f81165ca640 1 -- 172.21.15.100:0/2301399431 >> 172.21.15.100:0/2301399431 conn(0x7f8110066a00 msgr2=0x7f8110066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:16.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.244+0000 7f81165ca640 1 -- 172.21.15.100:0/2301399431 shutdown_connections 2024-03-29T11:45:16.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.244+0000 7f81165ca640 1 -- 172.21.15.100:0/2301399431 wait complete. 2024-03-29T11:45:16.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.244+0000 7f81165ca640 10 monclient: build_initial_monmap 2024-03-29T11:45:16.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.244+0000 7f81165ca640 10 monclient: monmap: 2024-03-29T11:45:16.260 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:16.260 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:16.261 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:45:16.262 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:45:16.262 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:16.262 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:16.262 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:16.262 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:45:16.262 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:45:16.262 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.245+0000 7f81165ca640 1 librados: starting msgr at 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.245+0000 7f81165ca640 1 librados: starting objecter 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.245+0000 7f81165ca640 1 Processor -- start 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.245+0000 7f81165ca640 1 -- start start 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.245+0000 7f81165ca640 1 librados: setting wanted keys 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.245+0000 7f81165ca640 1 librados: calling monclient init 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.245+0000 7f81165ca640 10 monclient: init 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f81165ca640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f81165ca640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f81165ca640 10 monclient(hunting): picked mon.noname-a con 0x7f81100a9d70 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f81165ca640 10 monclient(hunting): picked mon.noname-b con 0x7f811013f650 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f81165ca640 10 monclient(hunting): picked mon.noname-c con 0x7f81101457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f81165ca640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8110141a80 con 0x7f81100a9d70 2024-03-29T11:45:16.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f810f7fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f811013f650 0x7f8110134fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:58618/0 (socket says 172.21.15.100:58618) 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f810f7fe640 1 -- 172.21.15.100:0/1110647591 learned_addr learned my addr 172.21.15.100:0/1110647591 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8110141d60 con 0x7f811013f650 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811013ffb0 con 0x7f81101457f0 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f81165ca640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.246+0000 7f81165ca640 10 monclient(hunting): authenticate will time out at 1436.441990s 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 client.?.objecter ms_handle_connect 0x7f811013f650 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 client.?.objecter ms_handle_connect 0x7f81100a9d70 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907824277 0 0) 0x7f8110141d60 con 0x7f811013f650 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:16.264 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc004eb0 con 0x7f811013f650 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154984001 0 0) 0x7f8110141a80 con 0x7f81100a9d70 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8110141d60 con 0x7f81100a9d70 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 client.?.objecter ms_handle_connect 0x7f81101457f0 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1732690550 0 0) 0x7f80fc004eb0 con 0x7f811013f650 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.247+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8110141a80 con 0x7f811013f650 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8104001a10 con 0x7f811013f650 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:16.265 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient(hunting): dump: 2024-03-29T11:45:16.266 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:16.266 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:16.266 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:16.266 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:16.266 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:16.266 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:16.266 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:16.266 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:16.266 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:16.266 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1148851426 0 0) 0x7f8110141d60 con 0x7f81100a9d70 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80fc004eb0 con 0x7f81100a9d70 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8100001a20 con 0x7f81100a9d70 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient(hunting): dump: 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:16.267 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1176578399 0 0) 0x7f8110141a80 con 0x7f811013f650 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 >> v1:172.21.15.100:6790/0 conn(0x7f81101457f0 legacy=0x7f81101e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 >> v1:172.21.15.100:6789/0 conn(0x7f81100a9d70 legacy=0x7f81101424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 monclient: found mon.b 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8110142be0 con 0x7f811013f650 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient: _finish_auth 0 2024-03-29T11:45:16.268 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f81165ca640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f81165ca640 10 monclient: _renew_subs 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f81165ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8110140540 con 0x7f811013f650 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f81165ca640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f81165ca640 10 monclient: _renew_subs 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f81165ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8110145000 con 0x7f811013f650 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f81165ca640 1 librados: init done 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8104001b70 con 0x7f811013f650 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8104004c30 con 0x7f811013f650 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:16.269 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:16.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810e7fc640 4 set_mon_vals no callback set 2024-03-29T11:45:16.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.248+0000 7f810cff9640 10 monclient: dump: 2024-03-29T11:45:16.270 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:16.270 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:16.272 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:16.272 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:16.272 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:16.272 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:16.272 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:16.272 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:16.272 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:16.272 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:16.272 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.251+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f810401ac00 con 0x7f811013f650 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.251+0000 7f81165ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.251+0000 7f81165ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.251+0000 7f81165ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.251+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f0005080 con 0x7f811013f650 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.252+0000 7f810ffff640 10 monclient: get_auth_request con 0x7f80fc06dee0 auth_method 0 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.252+0000 7f810cff9640 10 client.4474.objecter ms_handle_connect 0x7f80fc06dee0 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.252+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f8104084c10 con 0x7f811013f650 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.252+0000 7f810cff9640 10 client.4474.objecter ms_dispatch 0x7f8110000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.252+0000 7f810cff9640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.252+0000 7f810cff9640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:16.273 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.252+0000 7f810cff9640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T11:45:16.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.260+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81040504f0 con 0x7f811013f650 2024-03-29T11:45:16.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.260+0000 7f810cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:16.274 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.260+0000 7f810cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:16.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.657+0000 7f81165ca640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:16.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.657+0000 7f81165ca640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:16.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.657+0000 7f81165ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:45:16.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.657+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80f0005330 con 0x7f811013f650 2024-03-29T11:45:16.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.662+0000 7f810cff9640 1 -- 172.21.15.100:0/1110647591 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f8104052420 con 0x7f811013f650 2024-03-29T11:45:16.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.662+0000 7f810cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:16.666 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.662+0000 7f810cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:16.667 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:45:16.667 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:16.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.666+0000 7f81165ca640 10 librados: watch_flush enter 2024-03-29T11:45:16.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.666+0000 7f81165ca640 10 librados: watch_flush exit 2024-03-29T11:45:16.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.666+0000 7f81165ca640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T11:45:16.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.666+0000 7f81165ca640 10 client.4474.objecter successfully canceled tick 2024-03-29T11:45:16.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.666+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 >> v1:172.21.15.100:6816/3220289657 conn(0x7f80fc06dee0 legacy=0x7f80fc0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:16.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.666+0000 7f81165ca640 10 monclient: shutdown 2024-03-29T11:45:16.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.666+0000 7f81165ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:16.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.666+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 >> v1:172.21.15.191:6789/0 conn(0x7f811013f650 legacy=0x7f8110134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:16.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.667+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 shutdown_connections 2024-03-29T11:45:16.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.667+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 >> 172.21.15.100:0/1110647591 conn(0x7f8110066a00 msgr2=0x7f81100b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:16.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.667+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 shutdown_connections 2024-03-29T11:45:16.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.668+0000 7f81165ca640 1 -- 172.21.15.100:0/1110647591 wait complete. 2024-03-29T11:45:16.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:16.668+0000 7f81165ca640 1 librados: shutdown 2024-03-29T11:45:16.687 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:23.495 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:23.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.641+0000 7faaf77cf640 10 monclient: get_monmap_and_config 2024-03-29T11:45:23.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.641+0000 7faaf77cf640 10 monclient: build_initial_monmap 2024-03-29T11:45:23.649 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.641+0000 7faaf77cf640 10 monclient: monmap: 2024-03-29T11:45:23.649 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:23.650 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:23.650 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:45:23.642732+0000 2024-03-29T11:45:23.650 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:45:23.642732+0000 2024-03-29T11:45:23.650 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:23.650 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:23.650 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:23.650 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:45:23.650 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:45:23.650 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:23.650 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.645+0000 7faaf77cf640 1 Processor -- start 2024-03-29T11:45:23.651 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.646+0000 7faaf77cf640 1 -- start start 2024-03-29T11:45:23.651 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.646+0000 7faaf77cf640 10 monclient: init 2024-03-29T11:45:23.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.646+0000 7faaf77cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:23.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.646+0000 7faaf77cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:23.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.646+0000 7faaf77cf640 10 monclient(hunting): picked mon.noname-a con 0x7faaf00b4100 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:23.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.647+0000 7faaf77cf640 10 monclient(hunting): picked mon.noname-b con 0x7faaf013ee90 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:23.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.647+0000 7faaf77cf640 10 monclient(hunting): picked mon.noname-c con 0x7faaf00af900 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:23.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.647+0000 7faaf77cf640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf00afd10 con 0x7faaf00b4100 2024-03-29T11:45:23.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.647+0000 7faaf77cf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf00af6f0 con 0x7faaf00af900 2024-03-29T11:45:23.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.647+0000 7faaf77cf640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf00aac80 con 0x7faaf013ee90 2024-03-29T11:45:23.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.647+0000 7faaf77cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:23.657 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.647+0000 7faaf77cf640 10 monclient(hunting): authenticate will time out at 1443.842139s 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.648+0000 7faaf4d6a640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7faaf013ee90 0x7faaf013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:34300/0 (socket says 172.21.15.100:34300) 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.648+0000 7faaf4d6a640 1 -- 172.21.15.100:0/3571555682 learned_addr learned my addr 172.21.15.100:0/3571555682 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.649+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659880033 0 0) 0x7faaf00aac80 con 0x7faaf013ee90 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.649+0000 7faae7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.649+0000 7faae7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.649+0000 7faae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.649+0000 7faae7fff640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.649+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faadc003520 con 0x7faaf013ee90 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393319093 0 0) 0x7faaf00afd10 con 0x7faaf00b4100 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:45:23.658 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaf00aac80 con 0x7faaf00b4100 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4008453156 0 0) 0x7faadc003520 con 0x7faaf013ee90 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 >> v1:172.21.15.191:6789/0 conn(0x7faaf00af900 legacy=0x7faaf014be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 >> v1:172.21.15.100:6789/0 conn(0x7faaf00b4100 legacy=0x7faaf00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 1 monclient: found mon.noname-b 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf00aaf60 con 0x7faaf013ee90 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: _finish_auth 0 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faaf77cf640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faaf77cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:23.659 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7faae0001650 con 0x7faaf013ee90 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: dump: 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:23.660 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faaf77cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faae00018a0 con 0x7faaf013ee90 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 1 -- 172.21.15.100:0/3571555682 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7faae0001c30 con 0x7faaf013ee90 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.650+0000 7faae7fff640 10 monclient: dump: 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:23.661 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:23.662 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:23.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.651+0000 7faaf77cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:23.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.651+0000 7faaf77cf640 4 set_mon_vals no callback set 2024-03-29T11:45:23.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.651+0000 7faaf77cf640 10 monclient: shutdown 2024-03-29T11:45:23.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.651+0000 7faaf77cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:23.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.651+0000 7faaf77cf640 1 -- 172.21.15.100:0/3571555682 >> v1:172.21.15.100:6790/0 conn(0x7faaf013ee90 legacy=0x7faaf013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:23.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.652+0000 7faaf77cf640 1 -- 172.21.15.100:0/3571555682 shutdown_connections 2024-03-29T11:45:23.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.652+0000 7faaf77cf640 1 -- 172.21.15.100:0/3571555682 >> 172.21.15.100:0/3571555682 conn(0x7faaf00b35e0 msgr2=0x7faaf00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:23.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.652+0000 7faaf77cf640 1 -- 172.21.15.100:0/3571555682 shutdown_connections 2024-03-29T11:45:23.662 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.652+0000 7faaf77cf640 1 -- 172.21.15.100:0/3571555682 wait complete. 2024-03-29T11:45:23.669 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.653+0000 7faaf77cf640 10 monclient: build_initial_monmap 2024-03-29T11:45:23.669 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.653+0000 7faaf77cf640 10 monclient: monmap: 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.653+0000 7faaf77cf640 1 librados: starting msgr at 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.653+0000 7faaf77cf640 1 librados: starting objecter 2024-03-29T11:45:23.670 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.654+0000 7faaf77cf640 1 Processor -- start 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.654+0000 7faaf77cf640 1 -- start start 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.654+0000 7faaf77cf640 1 librados: setting wanted keys 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.654+0000 7faaf77cf640 1 librados: calling monclient init 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.654+0000 7faaf77cf640 10 monclient: init 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.655+0000 7faaf77cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.655+0000 7faaf77cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.655+0000 7faaf77cf640 10 monclient(hunting): picked mon.noname-c con 0x7faaf00af900 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.655+0000 7faaf556b640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7faaf00af900 0x7faaf01d6510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:53124/0 (socket says 172.21.15.100:53124) 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.655+0000 7faaf556b640 1 -- 172.21.15.100:0/3319739957 learned_addr learned my addr 172.21.15.100:0/3319739957 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.655+0000 7faaf77cf640 10 monclient(hunting): picked mon.noname-a con 0x7faaf00b4100 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.656+0000 7faaf77cf640 10 monclient(hunting): picked mon.noname-b con 0x7faaf013ee90 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.656+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf006f670 con 0x7faaf00b4100 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.656+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf0141b00 con 0x7faaf013ee90 2024-03-29T11:45:23.671 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf0141de0 con 0x7faaf00af900 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faaf77cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faaf77cf640 10 monclient(hunting): authenticate will time out at 1443.852091s 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faae5ffb640 10 client.?.objecter ms_handle_connect 0x7faaf00af900 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faae5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faae5ffb640 10 client.?.objecter ms_handle_connect 0x7faaf013ee90 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faae5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560443240 0 0) 0x7faaf0141de0 con 0x7faaf00af900 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faae5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faae5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faae5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faae5ffb640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.657+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad4004eb0 con 0x7faaf00af900 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4103888094 0 0) 0x7faad4004eb0 con 0x7faaf00af900 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faad4005300 con 0x7faaf00af900 2024-03-29T11:45:23.672 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7faae8001d70 con 0x7faaf00af900 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 10 monclient(hunting): dump: 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:23.673 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3609704945 0 0) 0x7faad4005300 con 0x7faaf00af900 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 >> v1:172.21.15.191:6789/0 conn(0x7faaf013ee90 legacy=0x7faaf01de850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 >> v1:172.21.15.100:6789/0 conn(0x7faaf00b4100 legacy=0x7faaf0138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 1 monclient: found mon.c 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf01d6c20 con 0x7faaf00af900 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faae5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faaf77cf640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faaf77cf640 10 monclient: _renew_subs 2024-03-29T11:45:23.674 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faaf77cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.658+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf013ded0 con 0x7faaf00af900 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faae8002030 con 0x7faaf00af900 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faae5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7faae8003de0 con 0x7faaf00af900 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faae5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faae5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faae5ffb640 10 monclient: dump: 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:23.675 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faaf77cf640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faaf77cf640 10 monclient: _renew_subs 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faaf77cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf0072560 con 0x7faaf00af900 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faae77fe640 4 set_mon_vals no callback set 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.659+0000 7faaf77cf640 1 librados: init done 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.660+0000 7faaf77cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.660+0000 7faaf77cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.660+0000 7faaf77cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.660+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf00b29d0 con 0x7faaf00af900 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.661+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7faae8003fc0 con 0x7faaf00af900 2024-03-29T11:45:23.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.662+0000 7faaf4d6a640 10 monclient: get_auth_request con 0x7faad406e0d0 auth_method 0 2024-03-29T11:45:23.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.662+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7faae8084d40 con 0x7faaf00af900 2024-03-29T11:45:23.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.662+0000 7faae5ffb640 10 client.4622.objecter ms_dispatch 0x7faaf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:23.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.662+0000 7faae5ffb640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:23.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.662+0000 7faae5ffb640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:23.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.662+0000 7faae5ffb640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:45:23.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.663+0000 7faae5ffb640 10 client.4622.objecter ms_handle_connect 0x7faad406e0d0 2024-03-29T11:45:23.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.668+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faae8050620 con 0x7faaf00af900 2024-03-29T11:45:23.677 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.668+0000 7faae5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:23.678 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.668+0000 7faae5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:23.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.982+0000 7faaf77cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:23.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.982+0000 7faaf77cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:23.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.982+0000 7faaf77cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:23.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.982+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faaf0075f10 con 0x7faaf00af900 2024-03-29T11:45:23.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.982+0000 7faae5ffb640 1 -- 172.21.15.100:0/3319739957 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7faae8003c40 con 0x7faaf00af900 2024-03-29T11:45:23.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.982+0000 7faae5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:23.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.982+0000 7faae5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:23.987 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:45:23.988 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 10 librados: watch_flush enter 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 10 librados: watch_flush exit 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 >> v1:172.21.15.100:6816/3220289657 conn(0x7faad406e0d0 legacy=0x7faad4070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 10 monclient: shutdown 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 >> v1:172.21.15.100:6790/0 conn(0x7faaf00af900 legacy=0x7faaf01d6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 shutdown_connections 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 >> 172.21.15.100:0/3319739957 conn(0x7faaf00b35e0 msgr2=0x7faaf00b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 shutdown_connections 2024-03-29T11:45:23.991 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 1 -- 172.21.15.100:0/3319739957 wait complete. 2024-03-29T11:45:23.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:23.986+0000 7faaf77cf640 1 librados: shutdown 2024-03-29T11:45:24.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:30.815 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:30.968 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.962+0000 7ff26476c640 10 monclient: get_monmap_and_config 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.962+0000 7ff26476c640 10 monclient: build_initial_monmap 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.962+0000 7ff26476c640 10 monclient: monmap: 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:45:30.963489+0000 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:45:30.963489+0000 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:45:30.969 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:30.970 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.966+0000 7ff26476c640 1 Processor -- start 2024-03-29T11:45:30.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.967+0000 7ff26476c640 1 -- start start 2024-03-29T11:45:30.971 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.967+0000 7ff26476c640 10 monclient: init 2024-03-29T11:45:30.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.968+0000 7ff26476c640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:30.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.968+0000 7ff26476c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:30.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.968+0000 7ff26476c640 10 monclient(hunting): picked mon.noname-c con 0x7ff25c0a9ac0 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:30.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.968+0000 7ff26476c640 10 monclient(hunting): picked mon.noname-a con 0x7ff25c13ee90 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:30.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.968+0000 7ff26476c640 10 monclient(hunting): picked mon.noname-b con 0x7ff25c0af900 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:30.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.968+0000 7ff26476c640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c0afd10 con 0x7ff25c13ee90 2024-03-29T11:45:30.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.968+0000 7ff26476c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c0af6f0 con 0x7ff25c0a9ac0 2024-03-29T11:45:30.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.968+0000 7ff26476c640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c0aac80 con 0x7ff25c0af900 2024-03-29T11:45:30.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.968+0000 7ff26476c640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:30.973 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.968+0000 7ff26476c640 10 monclient(hunting): authenticate will time out at 1451.163862s 2024-03-29T11:45:30.975 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.971+0000 7ff262d09640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7ff25c0af900 0x7ff25c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:54722/0 (socket says 172.21.15.100:54722) 2024-03-29T11:45:30.976 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.971+0000 7ff262d09640 1 -- 172.21.15.100:0/1341034051 learned_addr learned my addr 172.21.15.100:0/1341034051 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:30.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.973+0000 7ff261506640 1 -- 172.21.15.100:0/1341034051 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748010111 0 0) 0x7ff25c0aac80 con 0x7ff25c0af900 2024-03-29T11:45:30.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.973+0000 7ff261506640 10 monclient(hunting): _negotiate 2024-03-29T11:45:30.977 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.973+0000 7ff261506640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:30.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.973+0000 7ff261506640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:30.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.973+0000 7ff261506640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:45:30.978 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.973+0000 7ff261506640 1 -- 172.21.15.100:0/1341034051 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff244003520 con 0x7ff25c0af900 2024-03-29T11:45:30.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 1 -- 172.21.15.100:0/1341034051 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3217416603 0 0) 0x7ff244003520 con 0x7ff25c0af900 2024-03-29T11:45:30.980 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 1 -- 172.21.15.100:0/1341034051 >> v1:172.21.15.191:6789/0 conn(0x7ff25c0a9ac0 legacy=0x7ff25c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 1 -- 172.21.15.100:0/1341034051 >> v1:172.21.15.100:6789/0 conn(0x7ff25c13ee90 legacy=0x7ff25c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: _finish_hunting 0 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 1 monclient: found mon.noname-b 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 1 -- 172.21.15.100:0/1341034051 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0aaf60 con 0x7ff25c0af900 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: _finish_auth 0 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: _check_auth_tickets 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 1 -- 172.21.15.100:0/1341034051 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff24c001a10 con 0x7ff25c0af900 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: dump: 2024-03-29T11:45:30.981 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 1 -- 172.21.15.100:0/1341034051 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff24c001b70 con 0x7ff25c0af900 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 1 -- 172.21.15.100:0/1341034051 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff24c001fc0 con 0x7ff25c0af900 2024-03-29T11:45:30.982 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.974+0000 7ff261506640 10 monclient: dump: 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:30.984 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.977+0000 7ff26476c640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:45:30.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.977+0000 7ff26476c640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:30.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.977+0000 7ff26476c640 4 set_mon_vals no callback set 2024-03-29T11:45:30.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.977+0000 7ff26476c640 10 monclient: shutdown 2024-03-29T11:45:30.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.977+0000 7ff26476c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:30.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.977+0000 7ff26476c640 1 -- 172.21.15.100:0/1341034051 >> v1:172.21.15.100:6790/0 conn(0x7ff25c0af900 legacy=0x7ff25c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:30.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.978+0000 7ff26476c640 1 -- 172.21.15.100:0/1341034051 shutdown_connections 2024-03-29T11:45:30.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.978+0000 7ff26476c640 1 -- 172.21.15.100:0/1341034051 >> 172.21.15.100:0/1341034051 conn(0x7ff25c0b35e0 msgr2=0x7ff25c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:30.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.979+0000 7ff26476c640 1 -- 172.21.15.100:0/1341034051 shutdown_connections 2024-03-29T11:45:30.985 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.979+0000 7ff26476c640 1 -- 172.21.15.100:0/1341034051 wait complete. 2024-03-29T11:45:30.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.980+0000 7ff26476c640 10 monclient: build_initial_monmap 2024-03-29T11:45:30.986 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.981+0000 7ff26476c640 10 monclient: monmap: 2024-03-29T11:45:30.986 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:30.986 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.981+0000 7ff26476c640 1 librados: starting msgr at 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.981+0000 7ff26476c640 1 librados: starting objecter 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.981+0000 7ff26476c640 1 Processor -- start 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.982+0000 7ff26476c640 1 -- start start 2024-03-29T11:45:30.987 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.982+0000 7ff26476c640 1 librados: setting wanted keys 2024-03-29T11:45:30.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.982+0000 7ff26476c640 1 librados: calling monclient init 2024-03-29T11:45:30.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.982+0000 7ff26476c640 10 monclient: init 2024-03-29T11:45:30.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.983+0000 7ff26476c640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:30.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.983+0000 7ff26476c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:30.988 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.983+0000 7ff26476c640 10 monclient(hunting): picked mon.noname-b con 0x7ff25c0a9ac0 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:30.990 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.986+0000 7ff26476c640 10 monclient(hunting): picked mon.noname-a con 0x7ff25c0af900 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:30.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.986+0000 7ff26476c640 10 monclient(hunting): picked mon.noname-c con 0x7ff25c13ee90 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:30.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.987+0000 7ff26476c640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c13a4d0 con 0x7ff25c0af900 2024-03-29T11:45:30.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.987+0000 7ff26476c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c1d6540 con 0x7ff25c0a9ac0 2024-03-29T11:45:30.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.987+0000 7ff26476c640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c06e8d0 con 0x7ff25c13ee90 2024-03-29T11:45:30.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.987+0000 7ff26476c640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:30.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.987+0000 7ff26476c640 10 monclient(hunting): authenticate will time out at 1451.182366s 2024-03-29T11:45:30.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.987+0000 7ff262d09640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7ff25c13ee90 0x7ff25c138cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:36952/0 (socket says 172.21.15.100:36952) 2024-03-29T11:45:30.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.987+0000 7ff262d09640 1 -- 172.21.15.100:0/1913683594 learned_addr learned my addr 172.21.15.100:0/1913683594 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:30.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.987+0000 7ff25affd640 10 client.?.objecter ms_handle_connect 0x7ff25c13ee90 2024-03-29T11:45:30.992 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.987+0000 7ff25affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:30.993 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.988+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820381122 0 0) 0x7ff25c06e8d0 con 0x7ff25c13ee90 2024-03-29T11:45:30.993 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.988+0000 7ff25affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:30.993 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.988+0000 7ff25affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:30.993 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.988+0000 7ff25affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:30.993 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.988+0000 7ff25affd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:45:30.993 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.988+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff248004eb0 con 0x7ff25c13ee90 2024-03-29T11:45:30.995 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 client.?.objecter ms_handle_connect 0x7ff25c0af900 2024-03-29T11:45:30.995 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:30.995 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2421870072 0 0) 0x7ff248004eb0 con 0x7ff25c13ee90 2024-03-29T11:45:30.995 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff248005300 con 0x7ff25c13ee90 2024-03-29T11:45:30.995 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678391678 0 0) 0x7ff25c13a4d0 con 0x7ff25c0af900 2024-03-29T11:45:30.995 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:30.995 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:30.995 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:30.995 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:45:30.995 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff248005760 con 0x7ff25c0af900 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 client.?.objecter ms_handle_connect 0x7ff25c0a9ac0 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915529239 0 0) 0x7ff25c1d6540 con 0x7ff25c0a9ac0 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff25c13a4d0 con 0x7ff25c0a9ac0 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff24c001e80 con 0x7ff25c13ee90 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): dump: 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:30.996 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:30.997 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:30.997 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:30.997 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:30.997 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:30.997 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:30.997 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:30.997 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:30.997 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:30.997 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.990+0000 7ff25affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 115322788 0 0) 0x7ff248005300 con 0x7ff25c13ee90 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 >> v1:172.21.15.191:6789/0 conn(0x7ff25c0a9ac0 legacy=0x7ff25c1d6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 >> v1:172.21.15.100:6789/0 conn(0x7ff25c0af900 legacy=0x7ff25c1367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 1 monclient: found mon.c 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c1d7130 con 0x7ff25c13ee90 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 10 monclient: _finish_auth 0 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:30.999 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff26476c640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff26476c640 10 monclient: _renew_subs 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff26476c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff26476c640 1 -- 172.21.15.100:0/1913683594 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25c140a70 con 0x7ff25c13ee90 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff24c004f40 con 0x7ff25c13ee90 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff25affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff26476c640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff26476c640 10 monclient: _renew_subs 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff26476c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff26476c640 1 -- 172.21.15.100:0/1913683594 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c1370b0 con 0x7ff25c13ee90 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff26476c640 1 librados: init done 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.991+0000 7ff260d05640 4 set_mon_vals no callback set 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.992+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff24c005250 con 0x7ff25c13ee90 2024-03-29T11:45:31.000 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.992+0000 7ff25affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.992+0000 7ff25affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.992+0000 7ff25affd640 10 monclient: dump: 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.993+0000 7ff26476c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:31.001 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.993+0000 7ff26476c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:31.002 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.993+0000 7ff26476c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:31.002 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.993+0000 7ff26476c640 1 -- 172.21.15.100:0/1913683594 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c1d63a0 con 0x7ff25c13ee90 2024-03-29T11:45:31.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.997+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7ff24c01b220 con 0x7ff25c13ee90 2024-03-29T11:45:31.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.998+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7ff24c0507c0 con 0x7ff25c13ee90 2024-03-29T11:45:31.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.998+0000 7ff25affd640 10 client.4628.objecter ms_dispatch 0x7ff25c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:31.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.998+0000 7ff25affd640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:31.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.998+0000 7ff25affd640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:31.003 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.998+0000 7ff25affd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:45:31.004 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.998+0000 7ff262508640 10 monclient: get_auth_request con 0x7ff2480725c0 auth_method 0 2024-03-29T11:45:31.004 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:30.998+0000 7ff25affd640 10 client.4628.objecter ms_handle_connect 0x7ff2480725c0 2024-03-29T11:45:31.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.003+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff24c0503f0 con 0x7ff25c13ee90 2024-03-29T11:45:31.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.003+0000 7ff25affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:31.007 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.003+0000 7ff25affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:31.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.336+0000 7ff26476c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:31.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.336+0000 7ff26476c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:31.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.336+0000 7ff26476c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:31.341 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.336+0000 7ff26476c640 1 -- 172.21.15.100:0/1913683594 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff25c1d63a0 con 0x7ff25c13ee90 2024-03-29T11:45:31.342 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:45:31.342 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:31.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.337+0000 7ff25affd640 1 -- 172.21.15.100:0/1913683594 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7ff25c1d63a0 con 0x7ff25c13ee90 2024-03-29T11:45:31.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.337+0000 7ff25affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:31.342 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.337+0000 7ff25affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:31.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.340+0000 7ff26476c640 10 librados: watch_flush enter 2024-03-29T11:45:31.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.341+0000 7ff26476c640 10 librados: watch_flush exit 2024-03-29T11:45:31.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.341+0000 7ff26476c640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:45:31.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.341+0000 7ff26476c640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:45:31.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.341+0000 7ff26476c640 1 -- 172.21.15.100:0/1913683594 >> v1:172.21.15.100:6816/3220289657 conn(0x7ff2480725c0 legacy=0x7ff248074a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:31.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.341+0000 7ff26476c640 10 monclient: shutdown 2024-03-29T11:45:31.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.341+0000 7ff26476c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:31.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.341+0000 7ff26476c640 1 -- 172.21.15.100:0/1913683594 >> v1:172.21.15.100:6790/0 conn(0x7ff25c13ee90 legacy=0x7ff25c138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:31.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.342+0000 7ff26476c640 1 -- 172.21.15.100:0/1913683594 shutdown_connections 2024-03-29T11:45:31.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.342+0000 7ff26476c640 1 -- 172.21.15.100:0/1913683594 >> 172.21.15.100:0/1913683594 conn(0x7ff25c0b35e0 msgr2=0x7ff25c0b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:31.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.342+0000 7ff26476c640 1 -- 172.21.15.100:0/1913683594 shutdown_connections 2024-03-29T11:45:31.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.342+0000 7ff26476c640 1 -- 172.21.15.100:0/1913683594 wait complete. 2024-03-29T11:45:31.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:31.342+0000 7ff26476c640 1 librados: shutdown 2024-03-29T11:45:31.366 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:38.174 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:38.326 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.322+0000 7f29248fb640 10 monclient: get_monmap_and_config 2024-03-29T11:45:38.326 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.322+0000 7f29248fb640 10 monclient: build_initial_monmap 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.322+0000 7f29248fb640 10 monclient: monmap: 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:45:38.323154+0000 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:45:38.323154+0000 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:45:38.327 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:38.328 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.323+0000 7f29248fb640 1 Processor -- start 2024-03-29T11:45:38.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.323+0000 7f29248fb640 1 -- start start 2024-03-29T11:45:38.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.323+0000 7f29248fb640 10 monclient: init 2024-03-29T11:45:38.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.324+0000 7f29248fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:38.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.324+0000 7f29248fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:38.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.324+0000 7f29248fb640 10 monclient(hunting): picked mon.noname-b con 0x7f2918006a70 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:38.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.324+0000 7f29248fb640 10 monclient(hunting): picked mon.noname-a con 0x7f291802b100 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:38.330 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.324+0000 7f29248fb640 10 monclient(hunting): picked mon.noname-c con 0x7f2918029c30 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:38.331 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.324+0000 7f29248fb640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291802a270 con 0x7f291802b100 2024-03-29T11:45:38.331 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.324+0000 7f29248fb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291802c440 con 0x7f2918029c30 2024-03-29T11:45:38.331 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.324+0000 7f29248fb640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29180c9440 con 0x7f2918006a70 2024-03-29T11:45:38.331 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.324+0000 7f29248fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:38.331 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.324+0000 7f29248fb640 10 monclient(hunting): authenticate will time out at 1458.519909s 2024-03-29T11:45:38.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.326+0000 7f291f7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f2918006a70 0x7f29180c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:52106/0 (socket says 172.21.15.100:52106) 2024-03-29T11:45:38.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.326+0000 7f291f7fe640 1 -- 172.21.15.100:0/2056679006 learned_addr learned my addr 172.21.15.100:0/2056679006 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:38.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.326+0000 7f291e7fc640 1 -- 172.21.15.100:0/2056679006 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864435520 0 0) 0x7f29180c9440 con 0x7f2918006a70 2024-03-29T11:45:38.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.327+0000 7f291e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:38.332 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.327+0000 7f291e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:38.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.327+0000 7f291e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:38.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.327+0000 7f291e7fc640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:45:38.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.327+0000 7f291e7fc640 1 -- 172.21.15.100:0/2056679006 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2910003520 con 0x7f2918006a70 2024-03-29T11:45:38.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.327+0000 7f291e7fc640 1 -- 172.21.15.100:0/2056679006 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544372960 0 0) 0x7f2910003520 con 0x7f2918006a70 2024-03-29T11:45:38.333 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.327+0000 7f291e7fc640 1 -- 172.21.15.100:0/2056679006 >> v1:172.21.15.191:6789/0 conn(0x7f2918029c30 legacy=0x7f29180d74a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.328+0000 7f291e7fc640 1 -- 172.21.15.100:0/2056679006 >> v1:172.21.15.100:6789/0 conn(0x7f291802b100 legacy=0x7f29180cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.328+0000 7f291e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.328+0000 7f291e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.328+0000 7f291e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.328+0000 7f291e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 1 -- 172.21.15.100:0/2056679006 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29180c95d0 con 0x7f2918006a70 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 1 -- 172.21.15.100:0/2056679006 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2914001690 con 0x7f2918006a70 2024-03-29T11:45:38.336 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:38.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:38.337 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 10 monclient: dump: 2024-03-29T11:45:38.337 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:38.337 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:38.337 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:38.337 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:38.337 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:38.337 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:38.337 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:38.337 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 1 -- 172.21.15.100:0/2056679006 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2914001920 con 0x7f2918006a70 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 1 -- 172.21.15.100:0/2056679006 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f2914001cf0 con 0x7f2918006a70 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.329+0000 7f291e7fc640 10 monclient: dump: 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:38.338 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.330+0000 7f29248fb640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.330+0000 7f29248fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.330+0000 7f29248fb640 4 set_mon_vals no callback set 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.331+0000 7f29248fb640 10 monclient: shutdown 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.331+0000 7f29248fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.331+0000 7f29248fb640 1 -- 172.21.15.100:0/2056679006 >> v1:172.21.15.100:6790/0 conn(0x7f2918006a70 legacy=0x7f29180c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.331+0000 7f29248fb640 1 -- 172.21.15.100:0/2056679006 shutdown_connections 2024-03-29T11:45:38.339 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.331+0000 7f29248fb640 1 -- 172.21.15.100:0/2056679006 >> 172.21.15.100:0/2056679006 conn(0x7f29180058f0 msgr2=0x7f2918005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:38.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.331+0000 7f29248fb640 1 -- 172.21.15.100:0/2056679006 shutdown_connections 2024-03-29T11:45:38.340 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.331+0000 7f29248fb640 1 -- 172.21.15.100:0/2056679006 wait complete. 2024-03-29T11:45:38.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.332+0000 7f29248fb640 10 monclient: build_initial_monmap 2024-03-29T11:45:38.345 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.332+0000 7f29248fb640 10 monclient: monmap: 2024-03-29T11:45:38.345 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:38.345 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:38.345 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:45:38.345 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.333+0000 7f29248fb640 1 librados: starting msgr at 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.333+0000 7f29248fb640 1 librados: starting objecter 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.333+0000 7f29248fb640 1 Processor -- start 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.333+0000 7f29248fb640 1 -- start start 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.333+0000 7f29248fb640 1 librados: setting wanted keys 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.333+0000 7f29248fb640 1 librados: calling monclient init 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.333+0000 7f29248fb640 10 monclient: init 2024-03-29T11:45:38.346 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.334+0000 7f29248fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.334+0000 7f29248fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.334+0000 7f29248fb640 10 monclient(hunting): picked mon.noname-a con 0x7f2918006a70 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.334+0000 7f29248fb640 10 monclient(hunting): picked mon.noname-c con 0x7f2918029c30 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.334+0000 7f291effd640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f2918029c30 0x7f2918160d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:53212/0 (socket says 172.21.15.100:53212) 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.335+0000 7f29248fb640 10 monclient(hunting): picked mon.noname-b con 0x7f29180c2e20 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.335+0000 7f29248fb640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291802ef80 con 0x7f2918006a70 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.335+0000 7f29248fb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2918160880 con 0x7f29180c2e20 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.335+0000 7f29248fb640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291802f1d0 con 0x7f2918029c30 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.335+0000 7f29248fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.335+0000 7f29248fb640 10 monclient(hunting): authenticate will time out at 1458.530667s 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.335+0000 7f291effd640 1 -- 172.21.15.100:0/3679873678 learned_addr learned my addr 172.21.15.100:0/3679873678 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.336+0000 7f290bfff640 10 client.?.objecter ms_handle_connect 0x7f2918029c30 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.336+0000 7f290bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:38.347 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.337+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723682478 0 0) 0x7f291802f1d0 con 0x7f2918029c30 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.337+0000 7f290bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.337+0000 7f290bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.337+0000 7f290bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.337+0000 7f290bfff640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.337+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2904004eb0 con 0x7f2918029c30 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1502951476 0 0) 0x7f2904004eb0 con 0x7f2918029c30 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f291802f1d0 con 0x7f2918029c30 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f290c0017d0 con 0x7f2918029c30 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 10 monclient(hunting): dump: 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:38.348 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 612793908 0 0) 0x7f291802f1d0 con 0x7f2918029c30 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 >> v1:172.21.15.191:6789/0 conn(0x7f29180c2e20 legacy=0x7f291816c1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 >> v1:172.21.15.100:6789/0 conn(0x7f2918006a70 legacy=0x7f29180c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:38.349 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 1 monclient: found mon.c 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29180cd080 con 0x7f2918029c30 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 10 monclient: _finish_auth 0 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f290bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f29248fb640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f29248fb640 10 monclient: _renew_subs 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f29248fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f29248fb640 1 -- 172.21.15.100:0/3679873678 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291815e3c0 con 0x7f2918029c30 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f29248fb640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f29248fb640 10 monclient: _renew_subs 2024-03-29T11:45:38.350 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f29248fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:38.351 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f29248fb640 1 -- 172.21.15.100:0/3679873678 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2918161600 con 0x7f2918029c30 2024-03-29T11:45:38.351 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.338+0000 7f29248fb640 1 librados: init done 2024-03-29T11:45:38.352 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.340+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f290c001930 con 0x7f2918029c30 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.340+0000 7f290bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.340+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f290c004a50 con 0x7f2918029c30 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.340+0000 7f290bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.340+0000 7f290bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.340+0000 7f290bfff640 10 monclient: dump: 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:38.353 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.340+0000 7f291dffb640 4 set_mon_vals no callback set 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.341+0000 7f29248fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.341+0000 7f29248fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.341+0000 7f29248fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.341+0000 7f29248fb640 1 -- 172.21.15.100:0/3679873678 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2900005080 con 0x7f2918029c30 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.343+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f290c01aa20 con 0x7f2918029c30 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.343+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f290c084a70 con 0x7f2918029c30 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.343+0000 7f290bfff640 10 client.4634.objecter ms_dispatch 0x7f2920000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.343+0000 7f290bfff640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:38.354 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.343+0000 7f290bfff640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:38.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.343+0000 7f290bfff640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:45:38.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.343+0000 7f291f7fe640 10 monclient: get_auth_request con 0x7f290406db70 auth_method 0 2024-03-29T11:45:38.355 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.344+0000 7f290bfff640 10 client.4634.objecter ms_handle_connect 0x7f290406db70 2024-03-29T11:45:38.356 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.351+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f290c050350 con 0x7f2918029c30 2024-03-29T11:45:38.356 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.351+0000 7f290bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:38.356 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.351+0000 7f290bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:38.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.691+0000 7f29248fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:38.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.691+0000 7f29248fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:38.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.691+0000 7f29248fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:38.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.691+0000 7f29248fb640 1 -- 172.21.15.100:0/3679873678 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2900005330 con 0x7f2918029c30 2024-03-29T11:45:38.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.692+0000 7f290bfff640 1 -- 172.21.15.100:0/3679873678 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f290c052280 con 0x7f2918029c30 2024-03-29T11:45:38.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.692+0000 7f290bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:38.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.692+0000 7f290bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:38.697 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:45:38.697 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:38.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.696+0000 7f29248fb640 10 librados: watch_flush enter 2024-03-29T11:45:38.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.696+0000 7f29248fb640 10 librados: watch_flush exit 2024-03-29T11:45:38.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.696+0000 7f29248fb640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:45:38.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.696+0000 7f29248fb640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:45:38.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.697+0000 7f29248fb640 1 -- 172.21.15.100:0/3679873678 >> v1:172.21.15.100:6816/3220289657 conn(0x7f290406db70 legacy=0x7f2904070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:38.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.697+0000 7f29248fb640 10 monclient: shutdown 2024-03-29T11:45:38.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.697+0000 7f29248fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:38.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.697+0000 7f29248fb640 1 -- 172.21.15.100:0/3679873678 >> v1:172.21.15.100:6790/0 conn(0x7f2918029c30 legacy=0x7f2918160d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:38.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.697+0000 7f29248fb640 1 -- 172.21.15.100:0/3679873678 shutdown_connections 2024-03-29T11:45:38.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.697+0000 7f29248fb640 1 -- 172.21.15.100:0/3679873678 >> 172.21.15.100:0/3679873678 conn(0x7f29180058f0 msgr2=0x7f29180c3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:38.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.697+0000 7f29248fb640 1 -- 172.21.15.100:0/3679873678 shutdown_connections 2024-03-29T11:45:38.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.698+0000 7f29248fb640 1 -- 172.21.15.100:0/3679873678 wait complete. 2024-03-29T11:45:38.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:38.701+0000 7f29248fb640 1 librados: shutdown 2024-03-29T11:45:38.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:45.528 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:45.675 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.671+0000 7f931bf97640 10 monclient: get_monmap_and_config 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.671+0000 7f931bf97640 10 monclient: build_initial_monmap 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.671+0000 7f931bf97640 10 monclient: monmap: 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:45:45.672347+0000 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:45:45.672347+0000 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:45:45.676 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:45.678 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.673+0000 7f931bf97640 1 Processor -- start 2024-03-29T11:45:45.678 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.673+0000 7f931bf97640 1 -- start start 2024-03-29T11:45:45.678 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.673+0000 7f931bf97640 10 monclient: init 2024-03-29T11:45:45.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.674+0000 7f931bf97640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:45.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.674+0000 7f931bf97640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:45.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.674+0000 7f931bf97640 10 monclient(hunting): picked mon.noname-a con 0x7f93140b4330 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:45.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.674+0000 7f931bf97640 10 monclient(hunting): picked mon.noname-c con 0x7f93141470b0 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:45.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.674+0000 7f931bf97640 10 monclient(hunting): picked mon.noname-b con 0x7f93141439b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:45.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.674+0000 7f931bf97640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931406f2a0 con 0x7f93140b4330 2024-03-29T11:45:45.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.674+0000 7f931bf97640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9314146b90 con 0x7f93141470b0 2024-03-29T11:45:45.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.674+0000 7f931bf97640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93140b2ca0 con 0x7f93141439b0 2024-03-29T11:45:45.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.674+0000 7f931bf97640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:45.679 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.674+0000 7f931bf97640 10 monclient(hunting): authenticate will time out at 1465.869622s 2024-03-29T11:45:45.682 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.677+0000 7f9319d33640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f93140b4330 0x7f93140b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:51217/0 (socket says 172.21.15.100:51217) 2024-03-29T11:45:45.682 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.677+0000 7f9319d33640 1 -- 172.21.15.100:0/4130622439 learned_addr learned my addr 172.21.15.100:0/4130622439 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:45.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786057629 0 0) 0x7f9314146b90 con 0x7f93141470b0 2024-03-29T11:45:45.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): _negotiate 2024-03-29T11:45:45.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:45.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:45.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:45:45.685 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9300003520 con 0x7f93141470b0 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740897127 0 0) 0x7f931406f2a0 con 0x7f93140b4330 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): _negotiate 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9314146b90 con 0x7f93140b4330 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747110383 0 0) 0x7f93140b2ca0 con 0x7f93141439b0 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): _negotiate 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:45:45.686 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f931406f2a0 con 0x7f93141439b0 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 554770626 0 0) 0x7f9300003520 con 0x7f93141470b0 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 >> v1:172.21.15.100:6790/0 conn(0x7f93141439b0 legacy=0x7f9314151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 >> v1:172.21.15.100:6789/0 conn(0x7f93140b4330 legacy=0x7f93140b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient: _finish_hunting 0 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 monclient: found mon.noname-c 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.678+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931406e0c0 con 0x7f93141470b0 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.679+0000 7f9318d31640 10 monclient: _finish_auth 0 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.679+0000 7f9318d31640 10 monclient: _check_auth_tickets 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.679+0000 7f9318d31640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.679+0000 7f931bf97640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.679+0000 7f931bf97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:45.687 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.679+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f9310001690 con 0x7f93141470b0 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.679+0000 7f9318d31640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.679+0000 7f9318d31640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.679+0000 7f9318d31640 10 monclient: dump: 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:45.688 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.679+0000 7f931bf97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.680+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93100018b0 con 0x7f93141470b0 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.680+0000 7f9318d31640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.680+0000 7f9318d31640 1 -- 172.21.15.100:0/4130622439 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f9310001c40 con 0x7f93141470b0 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.680+0000 7f9318d31640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.680+0000 7f9318d31640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.680+0000 7f9318d31640 10 monclient: dump: 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:45.689 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:45.690 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:45.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.680+0000 7f931bf97640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:45.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.680+0000 7f931bf97640 4 set_mon_vals no callback set 2024-03-29T11:45:45.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.681+0000 7f931bf97640 10 monclient: shutdown 2024-03-29T11:45:45.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.681+0000 7f931bf97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:45.690 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.681+0000 7f931bf97640 1 -- 172.21.15.100:0/4130622439 >> v1:172.21.15.191:6789/0 conn(0x7f93141470b0 legacy=0x7f93140b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:45.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.682+0000 7f931bf97640 1 -- 172.21.15.100:0/4130622439 shutdown_connections 2024-03-29T11:45:45.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.682+0000 7f931bf97640 1 -- 172.21.15.100:0/4130622439 >> 172.21.15.100:0/4130622439 conn(0x7f9314066a00 msgr2=0x7f9314066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:45.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.682+0000 7f931bf97640 1 -- 172.21.15.100:0/4130622439 shutdown_connections 2024-03-29T11:45:45.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.682+0000 7f931bf97640 1 -- 172.21.15.100:0/4130622439 wait complete. 2024-03-29T11:45:45.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.683+0000 7f931bf97640 10 monclient: build_initial_monmap 2024-03-29T11:45:45.694 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.683+0000 7f931bf97640 10 monclient: monmap: 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.684+0000 7f931bf97640 1 librados: starting msgr at 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.684+0000 7f931bf97640 1 librados: starting objecter 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.684+0000 7f931bf97640 1 Processor -- start 2024-03-29T11:45:45.695 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.684+0000 7f931bf97640 1 -- start start 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.684+0000 7f931bf97640 1 librados: setting wanted keys 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.684+0000 7f931bf97640 1 librados: calling monclient init 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.684+0000 7f931bf97640 10 monclient: init 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931bf97640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931bf97640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931bf97640 10 monclient(hunting): picked mon.noname-a con 0x7f93140b4330 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931bf97640 10 monclient(hunting): picked mon.noname-b con 0x7f93141439b0 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931bf97640 10 monclient(hunting): picked mon.noname-c con 0x7f93141470b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931bf97640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93140b5ce0 con 0x7f93140b4330 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931bf97640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931407cc70 con 0x7f93141439b0 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931bf97640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93141ea020 con 0x7f93141470b0 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931bf97640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931bf97640 10 monclient(hunting): authenticate will time out at 1465.880439s 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931a534640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f93141470b0 0x7f93141e9910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:51700/0 (socket says 172.21.15.100:51700) 2024-03-29T11:45:45.696 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f931a534640 1 -- 172.21.15.100:0/4003355549 learned_addr learned my addr 172.21.15.100:0/4003355549 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f930a7fc640 10 client.?.objecter ms_handle_connect 0x7f93141470b0 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f930a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558727621 0 0) 0x7f93141ea020 con 0x7f93141470b0 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f930a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f930a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f930a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f930a7fc640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.685+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f8004eb0 con 0x7f93141470b0 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.687+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678542088 0 0) 0x7f92f8004eb0 con 0x7f93141470b0 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.687+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93141ea020 con 0x7f93141470b0 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.687+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f930c001d10 con 0x7f93141470b0 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.687+0000 7f930a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.687+0000 7f930a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:45.697 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.687+0000 7f930a7fc640 10 monclient(hunting): dump: 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.687+0000 7f930a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.687+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3208832695 0 0) 0x7f93141ea020 con 0x7f93141470b0 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.687+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 >> v1:172.21.15.191:6789/0 conn(0x7f93141439b0 legacy=0x7f93141e9200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:45.698 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.689+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 >> v1:172.21.15.100:6789/0 conn(0x7f93140b4330 legacy=0x7f93140b6170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:45.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f930a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:45.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f930a7fc640 1 monclient: found mon.c 2024-03-29T11:45:45.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f930a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:45.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f930a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:45.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93141ec470 con 0x7f93141470b0 2024-03-29T11:45:45.701 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f930a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f930a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f930a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f931bf97640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f931bf97640 10 monclient: _renew_subs 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f931bf97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f931bf97640 1 -- 172.21.15.100:0/4003355549 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93140b7410 con 0x7f93141470b0 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f931bf97640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f931bf97640 10 monclient: _renew_subs 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f931bf97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f931bf97640 1 -- 172.21.15.100:0/4003355549 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93140b6a60 con 0x7f93141470b0 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.690+0000 7f931bf97640 1 librados: init done 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.692+0000 7f92fffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:45.702 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.692+0000 7f92fffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.692+0000 7f92fffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.692+0000 7f92fffff640 1 -- 172.21.15.100:0/4003355549 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9314075f10 con 0x7f93141470b0 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.693+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f930c001e70 con 0x7f93141470b0 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.693+0000 7f930a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.693+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f930c003de0 con 0x7f93141470b0 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.693+0000 7f930a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.693+0000 7f930a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.693+0000 7f930a7fc640 10 monclient: dump: 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:45.703 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:45.704 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:45.704 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:45.704 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:45.704 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:45.704 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.693+0000 7f930bfff640 4 set_mon_vals no callback set 2024-03-29T11:45:45.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.697+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f930c003f40 con 0x7f93141470b0 2024-03-29T11:45:45.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.698+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f930c084e60 con 0x7f93141470b0 2024-03-29T11:45:45.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.698+0000 7f930a7fc640 10 client.4640.objecter ms_dispatch 0x7f9314000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:45.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.698+0000 7f930a7fc640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:45.705 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.698+0000 7f930a7fc640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:45.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.698+0000 7f930a7fc640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:45:45.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.698+0000 7f9319d33640 10 monclient: get_auth_request con 0x7f92f806dc90 auth_method 0 2024-03-29T11:45:45.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.699+0000 7f930a7fc640 10 client.4640.objecter ms_handle_connect 0x7f92f806dc90 2024-03-29T11:45:45.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.702+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f930c050740 con 0x7f93141470b0 2024-03-29T11:45:45.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.702+0000 7f930a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:45.706 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:45.702+0000 7f930a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:46.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.038+0000 7f92fffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:46.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.038+0000 7f92fffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:46.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.038+0000 7f92fffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:45:46.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.038+0000 7f92fffff640 1 -- 172.21.15.100:0/4003355549 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93140b75a0 con 0x7f93141470b0 2024-03-29T11:45:46.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.039+0000 7f930a7fc640 1 -- 172.21.15.100:0/4003355549 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f930c052670 con 0x7f93141470b0 2024-03-29T11:45:46.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.039+0000 7f930a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:46.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.039+0000 7f930a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:46.044 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:45:46.045 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:46.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.043+0000 7f92fffff640 10 librados: watch_flush enter 2024-03-29T11:45:46.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.044+0000 7f92fffff640 10 librados: watch_flush exit 2024-03-29T11:45:46.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.044+0000 7f92fffff640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:45:46.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.044+0000 7f92fffff640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:45:46.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.044+0000 7f92fffff640 1 -- 172.21.15.100:0/4003355549 >> v1:172.21.15.100:6816/3220289657 conn(0x7f92f806dc90 legacy=0x7f92f8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:46.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.044+0000 7f92fffff640 10 monclient: shutdown 2024-03-29T11:45:46.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.044+0000 7f92fffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:46.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.044+0000 7f92fffff640 1 -- 172.21.15.100:0/4003355549 >> v1:172.21.15.100:6790/0 conn(0x7f93141470b0 legacy=0x7f93141e9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:46.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.044+0000 7f92fffff640 1 -- 172.21.15.100:0/4003355549 shutdown_connections 2024-03-29T11:45:46.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.044+0000 7f92fffff640 1 -- 172.21.15.100:0/4003355549 >> 172.21.15.100:0/4003355549 conn(0x7f9314066a00 msgr2=0x7f93140b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:46.051 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.047+0000 7f92fffff640 1 -- 172.21.15.100:0/4003355549 shutdown_connections 2024-03-29T11:45:46.051 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.047+0000 7f92fffff640 1 -- 172.21.15.100:0/4003355549 wait complete. 2024-03-29T11:45:46.052 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:46.047+0000 7f92fffff640 1 librados: shutdown 2024-03-29T11:45:46.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:52.876 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.019+0000 7fabc60d3640 10 monclient: get_monmap_and_config 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.019+0000 7fabc60d3640 10 monclient: build_initial_monmap 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.019+0000 7fabc60d3640 10 monclient: monmap: 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:45:53.020382+0000 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:45:53.020382+0000 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:45:53.024 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:45:53.025 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:53.025 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.020+0000 7fabc60d3640 1 Processor -- start 2024-03-29T11:45:53.025 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.020+0000 7fabc60d3640 1 -- start start 2024-03-29T11:45:53.025 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.020+0000 7fabc60d3640 10 monclient: init 2024-03-29T11:45:53.026 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabc60d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:53.026 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabc60d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:53.026 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabc60d3640 10 monclient(hunting): picked mon.noname-a con 0x7fabc00b4710 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:53.026 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabc60d3640 10 monclient(hunting): picked mon.noname-c con 0x7fabc00b3970 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:53.027 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabc60d3640 10 monclient(hunting): picked mon.noname-b con 0x7fabc013f390 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:53.027 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabc60d3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00af020 con 0x7fabc00b4710 2024-03-29T11:45:53.027 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabc60d3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00b0910 con 0x7fabc00b3970 2024-03-29T11:45:53.027 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabc60d3640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00b0bf0 con 0x7fabc013f390 2024-03-29T11:45:53.027 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabc60d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:53.027 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabc60d3640 10 monclient(hunting): authenticate will time out at 1473.216660s 2024-03-29T11:45:53.027 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabbeffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fabc00b3970 0x7fabc00af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:33614/0 (socket says 172.21.15.100:33614) 2024-03-29T11:45:53.027 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.021+0000 7fabbeffd640 1 -- 172.21.15.100:0/2509358299 learned_addr learned my addr 172.21.15.100:0/2509358299 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:53.029 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506986962 0 0) 0x7fabc00af020 con 0x7fabc00b4710 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba8003520 con 0x7fabc00b4710 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030798044 0 0) 0x7fabc00b0910 con 0x7fabc00b3970 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc00af020 con 0x7fabc00b3970 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171294524 0 0) 0x7fabc00b0bf0 con 0x7fabc013f390 2024-03-29T11:45:53.030 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.022+0000 7fabbe7fc640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc00b0910 con 0x7fabc013f390 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1498055668 0 0) 0x7faba8003520 con 0x7fabc00b4710 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 >> v1:172.21.15.100:6790/0 conn(0x7fabc013f390 legacy=0x7fabc013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 >> v1:172.21.15.191:6789/0 conn(0x7fabc00b3970 legacy=0x7fabc00af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 1 monclient: found mon.noname-a 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc00aeb00 con 0x7fabc00b4710 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:53.031 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fabb0001690 con 0x7fabc00b4710 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: dump: 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:53.032 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabc60d3640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabc60d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabb00017f0 con 0x7fabc00b4710 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 1 -- 172.21.15.100:0/2509358299 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fabb0001c00 con 0x7fabc00b4710 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabc60d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabc60d3640 4 set_mon_vals no callback set 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.023+0000 7fabbe7fc640 10 monclient: dump: 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:53.033 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.024+0000 7fabc60d3640 10 monclient: shutdown 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.024+0000 7fabc60d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.024+0000 7fabc60d3640 1 -- 172.21.15.100:0/2509358299 >> v1:172.21.15.100:6789/0 conn(0x7fabc00b4710 legacy=0x7fabc00b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.024+0000 7fabc60d3640 1 -- 172.21.15.100:0/2509358299 shutdown_connections 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.024+0000 7fabc60d3640 1 -- 172.21.15.100:0/2509358299 >> 172.21.15.100:0/2509358299 conn(0x7fabc0066a00 msgr2=0x7fabc0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.024+0000 7fabc60d3640 1 -- 172.21.15.100:0/2509358299 shutdown_connections 2024-03-29T11:45:53.034 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.025+0000 7fabc60d3640 1 -- 172.21.15.100:0/2509358299 wait complete. 2024-03-29T11:45:53.040 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.026+0000 7fabc60d3640 10 monclient: build_initial_monmap 2024-03-29T11:45:53.040 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.026+0000 7fabc60d3640 10 monclient: monmap: 2024-03-29T11:45:53.040 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:45:53.040 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:53.040 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:45:53.040 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:45:53.040 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.026+0000 7fabc60d3640 1 librados: starting msgr at 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.026+0000 7fabc60d3640 1 librados: starting objecter 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.026+0000 7fabc60d3640 1 Processor -- start 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.026+0000 7fabc60d3640 1 -- start start 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.026+0000 7fabc60d3640 1 librados: setting wanted keys 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.026+0000 7fabc60d3640 1 librados: calling monclient init 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.026+0000 7fabc60d3640 10 monclient: init 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.027+0000 7fabc60d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:53.041 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.027+0000 7fabc60d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.027+0000 7fabc60d3640 10 monclient(hunting): picked mon.noname-a con 0x7fabc00b3970 addr v1:172.21.15.100:6789/0 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7fabc60d3640 10 monclient(hunting): picked mon.noname-c con 0x7fabc00b4710 addr v1:172.21.15.100:6790/0 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7fabbf7fe640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7fabc00b3970 0x7fabc0136120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:57996/0 (socket says 172.21.15.100:57996) 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7fabbf7fe640 1 -- 172.21.15.100:0/3140957490 learned_addr learned my addr 172.21.15.100:0/3140957490 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 client.?.objecter ms_handle_connect 0x7fabc00b3970 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7fabc60d3640 10 monclient(hunting): picked mon.noname-b con 0x7fabc013f390 addr v1:172.21.15.191:6789/0 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc01371c0 con 0x7fabc00b3970 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc01374a0 con 0x7fabc013f390 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc0137780 con 0x7fabc00b4710 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7fabc60d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7fabc60d3640 10 monclient(hunting): authenticate will time out at 1473.223364s 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214228308 0 0) 0x7fabc01371c0 con 0x7fabc00b3970 2024-03-29T11:45:53.042 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabac004eb0 con 0x7fabc00b3970 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 client.?.objecter ms_handle_connect 0x7fabc013f390 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 client.?.objecter ms_handle_connect 0x7fabc00b4710 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238502908 0 0) 0x7fabc01374a0 con 0x7fabc013f390 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:45:53.043 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabac0051d0 con 0x7fabc013f390 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1778070010 0 0) 0x7fabac004eb0 con 0x7fabc00b3970 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabc01374a0 con 0x7fabc00b3970 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fabb0001b80 con 0x7fabc00b3970 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): dump: 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:53.044 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.028+0000 7faba7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2155321899 0 0) 0x7fabc01374a0 con 0x7fabc00b3970 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 >> v1:172.21.15.100:6790/0 conn(0x7fabc00b4710 legacy=0x7fabc0137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 >> v1:172.21.15.191:6789/0 conn(0x7fabc013f390 legacy=0x7fabc01e2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 1 monclient: found mon.a 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc0135650 con 0x7fabc00b3970 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 10 monclient: _finish_auth 0 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7faba7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:53.045 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7fabc60d3640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7fabc60d3640 10 monclient: _renew_subs 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7fabc60d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc01358a0 con 0x7fabc00b3970 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7fabc60d3640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7fabc60d3640 10 monclient: _renew_subs 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7fabc60d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc0142170 con 0x7fabc00b3970 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.029+0000 7fabc60d3640 1 librados: init done 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.030+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabb00011f0 con 0x7fabc00b3970 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.030+0000 7faba7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.030+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fabb0003a80 con 0x7fabc00b3970 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.030+0000 7faba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.030+0000 7faba7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:53.046 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.030+0000 7faba7fff640 10 monclient: dump: 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.030+0000 7fabbdffb640 4 set_mon_vals no callback set 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.030+0000 7fabc60d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.031+0000 7fabc60d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.031+0000 7fabc60d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:53.047 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.031+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba0005080 con 0x7fabc00b3970 2024-03-29T11:45:53.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.032+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fabb0003be0 con 0x7fabc00b3970 2024-03-29T11:45:53.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.033+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fabb0084ac0 con 0x7fabc00b3970 2024-03-29T11:45:53.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.033+0000 7faba7fff640 10 client.4533.objecter ms_dispatch 0x7fabc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:53.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.033+0000 7faba7fff640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:53.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.033+0000 7faba7fff640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:53.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.033+0000 7faba7fff640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-29T11:45:53.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.033+0000 7fabbeffd640 10 monclient: get_auth_request con 0x7fabac06ddb0 auth_method 0 2024-03-29T11:45:53.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.033+0000 7faba7fff640 10 client.4533.objecter ms_handle_connect 0x7fabac06ddb0 2024-03-29T11:45:53.048 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.041+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabb00501d0 con 0x7fabc00b3970 2024-03-29T11:45:53.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.041+0000 7faba7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:53.049 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.041+0000 7faba7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:53.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.404+0000 7fabc60d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:53.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.404+0000 7fabc60d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:53.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.404+0000 7fabc60d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:45:53.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.404+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faba0005330 con 0x7fabc00b3970 2024-03-29T11:45:53.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.406+0000 7faba7fff640 1 -- 172.21.15.100:0/3140957490 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fabb00522d0 con 0x7fabc00b3970 2024-03-29T11:45:53.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.406+0000 7faba7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:53.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.406+0000 7faba7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:53.411 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:45:53.411 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:53.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.409+0000 7fabc60d3640 10 librados: watch_flush enter 2024-03-29T11:45:53.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.410+0000 7fabc60d3640 10 librados: watch_flush exit 2024-03-29T11:45:53.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.410+0000 7fabc60d3640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-29T11:45:53.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.410+0000 7fabc60d3640 10 client.4533.objecter successfully canceled tick 2024-03-29T11:45:53.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.410+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 >> v1:172.21.15.100:6816/3220289657 conn(0x7fabac06ddb0 legacy=0x7fabac070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.410+0000 7fabc60d3640 10 monclient: shutdown 2024-03-29T11:45:53.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.410+0000 7fabc60d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:53.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.410+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 >> v1:172.21.15.100:6789/0 conn(0x7fabc00b3970 legacy=0x7fabc0136120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:53.415 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.410+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 shutdown_connections 2024-03-29T11:45:53.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.410+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 >> 172.21.15.100:0/3140957490 conn(0x7fabc0066a00 msgr2=0x7fabc01390b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:53.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.411+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 shutdown_connections 2024-03-29T11:45:53.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.411+0000 7fabc60d3640 1 -- 172.21.15.100:0/3140957490 wait complete. 2024-03-29T11:45:53.416 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:45:53.411+0000 7fabc60d3640 1 librados: shutdown 2024-03-29T11:45:53.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:00.238 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:00.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.380+0000 7fc3a4cd9640 10 monclient: get_monmap_and_config 2024-03-29T11:46:00.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.380+0000 7fc3a4cd9640 10 monclient: build_initial_monmap 2024-03-29T11:46:00.388 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.380+0000 7fc3a4cd9640 10 monclient: monmap: 2024-03-29T11:46:00.388 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:00.388 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:46:00.381808+0000 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:46:00.381808+0000 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.383+0000 7fc3a4cd9640 1 Processor -- start 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.383+0000 7fc3a4cd9640 1 -- start start 2024-03-29T11:46:00.389 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.383+0000 7fc3a4cd9640 10 monclient: init 2024-03-29T11:46:00.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.384+0000 7fc3a4cd9640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:00.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.384+0000 7fc3a4cd9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:46:00.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.384+0000 7fc3a4cd9640 10 monclient(hunting): picked mon.noname-c con 0x7fc3a00a9930 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:00.390 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.384+0000 7fc3a4cd9640 10 monclient(hunting): picked mon.noname-b con 0x7fc3a0149dc0 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:00.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.387+0000 7fc3a4cd9640 10 monclient(hunting): picked mon.noname-a con 0x7fc3a014ff40 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:00.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.387+0000 7fc3a4cd9640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a00aae60 con 0x7fc3a014ff40 2024-03-29T11:46:00.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.387+0000 7fc3a4cd9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a00b4310 con 0x7fc3a00a9930 2024-03-29T11:46:00.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.387+0000 7fc3a4cd9640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a00afdd0 con 0x7fc3a0149dc0 2024-03-29T11:46:00.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.387+0000 7fc3a4cd9640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:00.392 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.387+0000 7fc3a4cd9640 10 monclient(hunting): authenticate will time out at 1480.582431s 2024-03-29T11:46:00.394 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.387+0000 7fc39dd9b640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fc3a0149dc0 0x7fc3a014a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:43696/0 (socket says 172.21.15.100:43696) 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39dd9b640 1 -- 172.21.15.100:0/397280061 learned_addr learned my addr 172.21.15.100:0/397280061 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818534824 0 0) 0x7fc3a00afdd0 con 0x7fc3a0149dc0 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc388003520 con 0x7fc3a0149dc0 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550198232 0 0) 0x7fc3a00aae60 con 0x7fc3a014ff40 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:00.395 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a00afdd0 con 0x7fc3a014ff40 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 553629557 0 0) 0x7fc388003520 con 0x7fc3a0149dc0 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 >> v1:172.21.15.191:6789/0 conn(0x7fc3a00a9930 legacy=0x7fc3a0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 >> v1:172.21.15.100:6789/0 conn(0x7fc3a014ff40 legacy=0x7fc3a0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 1 monclient: found mon.noname-b 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a014a950 con 0x7fc3a0149dc0 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient: _finish_auth 0 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc398001990 con 0x7fc3a0149dc0 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:00.396 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.388+0000 7fc39d59a640 10 monclient: dump: 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.389+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc398001af0 con 0x7fc3a0149dc0 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.389+0000 7fc39d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:00.397 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.389+0000 7fc39d59a640 1 -- 172.21.15.100:0/397280061 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc398001f00 con 0x7fc3a0149dc0 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.389+0000 7fc39d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.389+0000 7fc39d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.389+0000 7fc39d59a640 10 monclient: dump: 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:00.398 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.389+0000 7fc3a4cd9640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:46:00.399 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.389+0000 7fc3a4cd9640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:00.399 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.389+0000 7fc3a4cd9640 4 set_mon_vals no callback set 2024-03-29T11:46:00.399 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.390+0000 7fc3a4cd9640 10 monclient: shutdown 2024-03-29T11:46:00.399 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.390+0000 7fc3a4cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:00.399 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.390+0000 7fc3a4cd9640 1 -- 172.21.15.100:0/397280061 >> v1:172.21.15.100:6790/0 conn(0x7fc3a0149dc0 legacy=0x7fc3a014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.390+0000 7fc3a4cd9640 1 -- 172.21.15.100:0/397280061 shutdown_connections 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.390+0000 7fc3a4cd9640 1 -- 172.21.15.100:0/397280061 >> 172.21.15.100:0/397280061 conn(0x7fc3a00b35e0 msgr2=0x7fc3a00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.391+0000 7fc3a4cd9640 1 -- 172.21.15.100:0/397280061 shutdown_connections 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.391+0000 7fc3a4cd9640 1 -- 172.21.15.100:0/397280061 wait complete. 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.392+0000 7fc3a4cd9640 10 monclient: build_initial_monmap 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.392+0000 7fc3a4cd9640 10 monclient: monmap: 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:00.405 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.392+0000 7fc3a4cd9640 1 librados: starting msgr at 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.392+0000 7fc3a4cd9640 1 librados: starting objecter 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.392+0000 7fc3a4cd9640 1 Processor -- start 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.392+0000 7fc3a4cd9640 1 -- start start 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.392+0000 7fc3a4cd9640 1 librados: setting wanted keys 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.392+0000 7fc3a4cd9640 1 librados: calling monclient init 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.392+0000 7fc3a4cd9640 10 monclient: init 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.393+0000 7fc3a4cd9640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.393+0000 7fc3a4cd9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:46:00.406 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.393+0000 7fc3a4cd9640 10 monclient(hunting): picked mon.noname-a con 0x7fc3a00a9930 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.393+0000 7fc3a4cd9640 10 monclient(hunting): picked mon.noname-b con 0x7fc3a0149dc0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.394+0000 7fc3a4cd9640 10 monclient(hunting): picked mon.noname-c con 0x7fc3a014ff40 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.394+0000 7fc3a4cd9640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a006e9b0 con 0x7fc3a00a9930 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.394+0000 7fc3a4cd9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a00716b0 con 0x7fc3a0149dc0 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.394+0000 7fc39ed9d640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fc3a014ff40 0x7fc3a01ef1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:44328/0 (socket says 172.21.15.100:44328) 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.394+0000 7fc39ed9d640 1 -- 172.21.15.100:0/486945121 learned_addr learned my addr 172.21.15.100:0/486945121 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.394+0000 7fc38effd640 10 client.?.objecter ms_handle_connect 0x7fc3a014ff40 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.394+0000 7fc38effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.394+0000 7fc3a4cd9640 1 -- 172.21.15.100:0/486945121 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a006ec20 con 0x7fc3a014ff40 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.394+0000 7fc3a4cd9640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.394+0000 7fc3a4cd9640 10 monclient(hunting): authenticate will time out at 1480.589899s 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.395+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702657158 0 0) 0x7fc3a006ec20 con 0x7fc3a014ff40 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.395+0000 7fc38effd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:00.407 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.395+0000 7fc38effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.395+0000 7fc38effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.395+0000 7fc38effd640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.395+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc380004eb0 con 0x7fc3a014ff40 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.396+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1758416313 0 0) 0x7fc380004eb0 con 0x7fc3a014ff40 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.396+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3a006ec20 con 0x7fc3a014ff40 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.396+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc394001890 con 0x7fc3a014ff40 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.396+0000 7fc38effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.396+0000 7fc38effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.396+0000 7fc38effd640 10 monclient(hunting): dump: 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:00.408 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.396+0000 7fc38effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.396+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1301675208 0 0) 0x7fc3a006ec20 con 0x7fc3a014ff40 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.396+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 >> v1:172.21.15.191:6789/0 conn(0x7fc3a0149dc0 legacy=0x7fc3a00ac750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 >> v1:172.21.15.100:6789/0 conn(0x7fc3a00a9930 legacy=0x7fc3a0070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc38effd640 10 monclient: _finish_hunting 0 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc38effd640 1 monclient: found mon.c 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc38effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc38effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a006ef00 con 0x7fc3a014ff40 2024-03-29T11:46:00.409 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc38effd640 10 monclient: _finish_auth 0 2024-03-29T11:46:00.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc38effd640 10 monclient: _check_auth_tickets 2024-03-29T11:46:00.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc38effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:00.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc3a4cd9640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:46:00.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc3a4cd9640 10 monclient: _renew_subs 2024-03-29T11:46:00.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc3a4cd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:00.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.398+0000 7fc3a4cd9640 1 -- 172.21.15.100:0/486945121 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a006f1e0 con 0x7fc3a014ff40 2024-03-29T11:46:00.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.399+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc394001a60 con 0x7fc3a014ff40 2024-03-29T11:46:00.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.399+0000 7fc38effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:00.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.399+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fc3940038c0 con 0x7fc3a014ff40 2024-03-29T11:46:00.410 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.399+0000 7fc39cd99640 4 set_mon_vals no callback set 2024-03-29T11:46:00.411 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.400+0000 7fc3a4cd9640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:00.411 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.400+0000 7fc3a4cd9640 10 monclient: _renew_subs 2024-03-29T11:46:00.411 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.400+0000 7fc3a4cd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:00.411 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.400+0000 7fc3a4cd9640 1 -- 172.21.15.100:0/486945121 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a006f8e0 con 0x7fc3a014ff40 2024-03-29T11:46:00.411 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.400+0000 7fc3a4cd9640 1 librados: init done 2024-03-29T11:46:00.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.401+0000 7fc38effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:00.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.401+0000 7fc38effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:00.412 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.401+0000 7fc38effd640 10 monclient: dump: 2024-03-29T11:46:00.412 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.401+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fc394003a20 con 0x7fc3a014ff40 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.401+0000 7fc38cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.401+0000 7fc38cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.401+0000 7fc38cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:00.413 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.401+0000 7fc38cff9640 1 -- 172.21.15.100:0/486945121 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a0139fe0 con 0x7fc3a014ff40 2024-03-29T11:46:00.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.402+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fc394084a80 con 0x7fc3a014ff40 2024-03-29T11:46:00.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.402+0000 7fc38effd640 10 client.4652.objecter ms_dispatch 0x7fc3a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:00.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.402+0000 7fc38effd640 3 client.4652.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:00.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.402+0000 7fc38effd640 3 client.4652.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:00.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.402+0000 7fc38effd640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-29T11:46:00.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.402+0000 7fc39e59c640 10 monclient: get_auth_request con 0x7fc38006dc90 auth_method 0 2024-03-29T11:46:00.414 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.402+0000 7fc38effd640 10 client.4652.objecter ms_handle_connect 0x7fc38006dc90 2024-03-29T11:46:00.417 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.413+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3940501b0 con 0x7fc3a014ff40 2024-03-29T11:46:00.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.413+0000 7fc38effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:00.418 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.413+0000 7fc38effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:00.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.774+0000 7fc38cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:00.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.774+0000 7fc38cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:00.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.774+0000 7fc38cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:00.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.774+0000 7fc38cff9640 1 -- 172.21.15.100:0/486945121 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3a0075f10 con 0x7fc3a014ff40 2024-03-29T11:46:00.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.775+0000 7fc38effd640 1 -- 172.21.15.100:0/486945121 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fc3940522b0 con 0x7fc3a014ff40 2024-03-29T11:46:00.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.775+0000 7fc38effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:00.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.775+0000 7fc38effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:00.780 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:46:00.780 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:00.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.778+0000 7fc38cff9640 10 librados: watch_flush enter 2024-03-29T11:46:00.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.778+0000 7fc38cff9640 10 librados: watch_flush exit 2024-03-29T11:46:00.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.778+0000 7fc38cff9640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-29T11:46:00.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.778+0000 7fc38cff9640 10 client.4652.objecter successfully canceled tick 2024-03-29T11:46:00.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.779+0000 7fc38cff9640 1 -- 172.21.15.100:0/486945121 >> v1:172.21.15.100:6816/3220289657 conn(0x7fc38006dc90 legacy=0x7fc380070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:00.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.779+0000 7fc38cff9640 10 monclient: shutdown 2024-03-29T11:46:00.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.779+0000 7fc38cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:00.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.779+0000 7fc38cff9640 1 -- 172.21.15.100:0/486945121 >> v1:172.21.15.100:6790/0 conn(0x7fc3a014ff40 legacy=0x7fc3a01ef1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:00.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.779+0000 7fc38cff9640 1 -- 172.21.15.100:0/486945121 shutdown_connections 2024-03-29T11:46:00.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.779+0000 7fc38cff9640 1 -- 172.21.15.100:0/486945121 >> 172.21.15.100:0/486945121 conn(0x7fc3a00b35e0 msgr2=0x7fc3a00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:00.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.779+0000 7fc38cff9640 1 -- 172.21.15.100:0/486945121 shutdown_connections 2024-03-29T11:46:00.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.779+0000 7fc38cff9640 1 -- 172.21.15.100:0/486945121 wait complete. 2024-03-29T11:46:00.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:00.779+0000 7fc38cff9640 1 librados: shutdown 2024-03-29T11:46:00.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:07.607 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.770+0000 7fd1a585f640 10 monclient: get_monmap_and_config 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.770+0000 7fd1a585f640 10 monclient: build_initial_monmap 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.770+0000 7fd1a585f640 10 monclient: monmap: 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:46:07.771154+0000 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:46:07.771154+0000 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:46:07.775 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:46:07.776 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:07.776 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.772+0000 7fd1a585f640 1 Processor -- start 2024-03-29T11:46:07.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.773+0000 7fd1a585f640 1 -- start start 2024-03-29T11:46:07.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.773+0000 7fd1a585f640 10 monclient: init 2024-03-29T11:46:07.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.774+0000 7fd1a585f640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:07.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.774+0000 7fd1a585f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:46:07.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.774+0000 7fd1a585f640 10 monclient(hunting): picked mon.noname-c con 0x7fd198006ec0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:07.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.774+0000 7fd1a585f640 10 monclient(hunting): picked mon.noname-a con 0x7fd198033530 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:07.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.774+0000 7fd1a585f640 10 monclient(hunting): picked mon.noname-b con 0x7fd1980cfcb0 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:07.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.774+0000 7fd1a585f640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1980cf110 con 0x7fd198033530 2024-03-29T11:46:07.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.774+0000 7fd1a585f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd19802c1d0 con 0x7fd198006ec0 2024-03-29T11:46:07.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.774+0000 7fd1a585f640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1980cbb90 con 0x7fd1980cfcb0 2024-03-29T11:46:07.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.774+0000 7fd1a585f640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:07.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.774+0000 7fd1a585f640 10 monclient(hunting): authenticate will time out at 1487.969195s 2024-03-29T11:46:07.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.777+0000 7fd1a505e640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fd1980cfcb0 0x7fd1980d74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:51982/0 (socket says 172.21.15.100:51982) 2024-03-29T11:46:07.782 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.777+0000 7fd1a505e640 1 -- 172.21.15.100:0/3160708044 learned_addr learned my addr 172.21.15.100:0/3160708044 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.777+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515913257 0 0) 0x7fd19802c1d0 con 0x7fd198006ec0 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.777+0000 7fd19f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.777+0000 7fd19f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.777+0000 7fd19f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.777+0000 7fd19f7fe640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.777+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd190003520 con 0x7fd198006ec0 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165880273 0 0) 0x7fd1980cbb90 con 0x7fd1980cfcb0 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:46:07.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd19802c1d0 con 0x7fd1980cfcb0 2024-03-29T11:46:07.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259763018 0 0) 0x7fd1980cf110 con 0x7fd198033530 2024-03-29T11:46:07.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1980cbb90 con 0x7fd198033530 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.778+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 92756193 0 0) 0x7fd190003520 con 0x7fd198006ec0 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 >> v1:172.21.15.100:6790/0 conn(0x7fd1980cfcb0 legacy=0x7fd1980d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 >> v1:172.21.15.100:6789/0 conn(0x7fd198033530 legacy=0x7fd1980cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd198029c30 con 0x7fd198006ec0 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:46:07.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd1a585f640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd1a585f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd18c001610 con 0x7fd198006ec0 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: dump: 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:07.786 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd1a585f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd18c001770 con 0x7fd198006ec0 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 1 -- 172.21.15.100:0/3160708044 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd18c001bc0 con 0x7fd198006ec0 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.779+0000 7fd19f7fe640 10 monclient: dump: 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:07.787 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:07.788 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:07.788 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:07.788 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:07.788 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:07.788 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:07.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.781+0000 7fd1a585f640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:07.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.781+0000 7fd1a585f640 4 set_mon_vals no callback set 2024-03-29T11:46:07.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.782+0000 7fd1a585f640 10 monclient: shutdown 2024-03-29T11:46:07.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.782+0000 7fd1a585f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:07.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.782+0000 7fd1a585f640 1 -- 172.21.15.100:0/3160708044 >> v1:172.21.15.191:6789/0 conn(0x7fd198006ec0 legacy=0x7fd198032e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:07.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.783+0000 7fd1a585f640 1 -- 172.21.15.100:0/3160708044 shutdown_connections 2024-03-29T11:46:07.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.783+0000 7fd1a585f640 1 -- 172.21.15.100:0/3160708044 >> 172.21.15.100:0/3160708044 conn(0x7fd198005d40 msgr2=0x7fd198006150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:07.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.783+0000 7fd1a585f640 1 -- 172.21.15.100:0/3160708044 shutdown_connections 2024-03-29T11:46:07.789 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.783+0000 7fd1a585f640 1 -- 172.21.15.100:0/3160708044 wait complete. 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.784+0000 7fd1a585f640 10 monclient: build_initial_monmap 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.784+0000 7fd1a585f640 10 monclient: monmap: 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.784+0000 7fd1a585f640 1 librados: starting msgr at 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.784+0000 7fd1a585f640 1 librados: starting objecter 2024-03-29T11:46:07.790 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.784+0000 7fd1a585f640 1 Processor -- start 2024-03-29T11:46:07.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.784+0000 7fd1a585f640 1 -- start start 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.784+0000 7fd1a585f640 1 librados: setting wanted keys 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.784+0000 7fd1a585f640 1 librados: calling monclient init 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.784+0000 7fd1a585f640 10 monclient: init 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.785+0000 7fd1a585f640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.785+0000 7fd1a585f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.785+0000 7fd1a585f640 10 monclient(hunting): picked mon.noname-c con 0x7fd198006ec0 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.785+0000 7fd1a585f640 10 monclient(hunting): picked mon.noname-a con 0x7fd1980cfcb0 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.785+0000 7fd1a585f640 10 monclient(hunting): picked mon.noname-b con 0x7fd198161810 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.785+0000 7fd1a585f640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd198031190 con 0x7fd1980cfcb0 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.785+0000 7fd1a585f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1980a9b20 con 0x7fd198161810 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.785+0000 7fd1a585f640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1980a9d90 con 0x7fd198006ec0 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.785+0000 7fd1a585f640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.785+0000 7fd1a585f640 10 monclient(hunting): authenticate will time out at 1487.980930s 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd1a505e640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd198161810 0x7fd1980c6bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:44319/0 (socket says 172.21.15.100:44319) 2024-03-29T11:46:07.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd1a505e640 1 -- 172.21.15.100:0/4093556275 learned_addr learned my addr 172.21.15.100:0/4093556275 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:07.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd19d7fa640 10 client.?.objecter ms_handle_connect 0x7fd198161810 2024-03-29T11:46:07.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd19d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:07.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600859406 0 0) 0x7fd1980a9b20 con 0x7fd198161810 2024-03-29T11:46:07.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd19d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:46:07.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd19d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:07.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd19d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:07.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd19d7fa640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:46:07.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184004eb0 con 0x7fd198161810 2024-03-29T11:46:07.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd19d7fa640 10 client.?.objecter ms_handle_connect 0x7fd198006ec0 2024-03-29T11:46:07.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.789+0000 7fd19d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:07.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.792+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1391170560 0 0) 0x7fd184004eb0 con 0x7fd198161810 2024-03-29T11:46:07.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.792+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1980a9b20 con 0x7fd198161810 2024-03-29T11:46:07.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.792+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353540512 0 0) 0x7fd1980a9d90 con 0x7fd198006ec0 2024-03-29T11:46:07.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:46:07.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:07.799 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd184004eb0 con 0x7fd198006ec0 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd1a004fdd0 con 0x7fd198161810 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient(hunting): dump: 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:07.800 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1839102537 0 0) 0x7fd1980a9b20 con 0x7fd198161810 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 >> v1:172.21.15.100:6790/0 conn(0x7fd198006ec0 legacy=0x7fd198031620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 >> v1:172.21.15.100:6789/0 conn(0x7fd1980cfcb0 legacy=0x7fd1980c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 1 monclient: found mon.b 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1980c72e0 con 0x7fd198161810 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd1a585f640 5 monclient: authenticate success, global_id 4504 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd1a585f640 10 monclient: _renew_subs 2024-03-29T11:46:07.801 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd1a585f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd1a585f640 1 -- 172.21.15.100:0/4093556275 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd198030c20 con 0x7fd198161810 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd1a585f640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd1a585f640 10 monclient: _renew_subs 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd1a585f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd1a585f640 1 -- 172.21.15.100:0/4093556275 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd198030ed0 con 0x7fd198161810 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd1a585f640 1 librados: init done 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1a004f400 con 0x7fd198161810 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fd1a0075f10 con 0x7fd198161810 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.793+0000 7fd19effd640 4 set_mon_vals no callback set 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.794+0000 7fd19d7fa640 10 monclient: dump: 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:07.802 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:07.803 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:07.803 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:07.803 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:07.803 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:07.803 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:07.803 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:07.803 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:07.803 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:07.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.796+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fd1a0067bf0 con 0x7fd198161810 2024-03-29T11:46:07.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.797+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7fd1a00ff400 con 0x7fd198161810 2024-03-29T11:46:07.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.797+0000 7fd19d7fa640 10 client.4504.objecter ms_dispatch 0x7fd1a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:07.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.797+0000 7fd19d7fa640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:07.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.797+0000 7fd19d7fa640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:07.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.797+0000 7fd19d7fa640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-29T11:46:07.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.797+0000 7fd1a485d640 10 monclient: get_auth_request con 0x7fd18406dba0 auth_method 0 2024-03-29T11:46:07.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.797+0000 7fd19d7fa640 10 client.4504.objecter ms_handle_connect 0x7fd18406dba0 2024-03-29T11:46:07.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.799+0000 7fd1a585f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:07.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.799+0000 7fd1a585f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:07.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.799+0000 7fd1a585f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:46:07.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.799+0000 7fd1a585f640 1 -- 172.21.15.100:0/4093556275 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd180005080 con 0x7fd198161810 2024-03-29T11:46:07.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.806+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1a00cace0 con 0x7fd198161810 2024-03-29T11:46:07.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.806+0000 7fd19d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:07.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:07.806+0000 7fd19d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:08.141 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.136+0000 7fd1a585f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:08.141 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.136+0000 7fd1a585f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:08.141 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.136+0000 7fd1a585f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:46:08.141 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.136+0000 7fd1a585f640 1 -- 172.21.15.100:0/4093556275 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd180005330 con 0x7fd198161810 2024-03-29T11:46:08.141 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.136+0000 7fd19d7fa640 1 -- 172.21.15.100:0/4093556275 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7fd1a00cace0 con 0x7fd198161810 2024-03-29T11:46:08.141 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.136+0000 7fd19d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:08.141 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.136+0000 7fd19d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:08.143 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:46:08.144 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:08.147 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.142+0000 7fd1a585f640 10 librados: watch_flush enter 2024-03-29T11:46:08.147 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.142+0000 7fd1a585f640 10 librados: watch_flush exit 2024-03-29T11:46:08.147 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.142+0000 7fd1a585f640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-29T11:46:08.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.142+0000 7fd1a585f640 10 client.4504.objecter successfully canceled tick 2024-03-29T11:46:08.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.142+0000 7fd1a585f640 1 -- 172.21.15.100:0/4093556275 >> v1:172.21.15.100:6816/3220289657 conn(0x7fd18406dba0 legacy=0x7fd184070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:08.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.142+0000 7fd1a585f640 10 monclient: shutdown 2024-03-29T11:46:08.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.142+0000 7fd1a585f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:08.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.142+0000 7fd1a585f640 1 -- 172.21.15.100:0/4093556275 >> v1:172.21.15.191:6789/0 conn(0x7fd198161810 legacy=0x7fd1980c6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:08.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.143+0000 7fd1a585f640 1 -- 172.21.15.100:0/4093556275 shutdown_connections 2024-03-29T11:46:08.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.143+0000 7fd1a585f640 1 -- 172.21.15.100:0/4093556275 >> 172.21.15.100:0/4093556275 conn(0x7fd198005d40 msgr2=0x7fd198037c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:08.148 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.143+0000 7fd1a585f640 1 -- 172.21.15.100:0/4093556275 shutdown_connections 2024-03-29T11:46:08.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.143+0000 7fd1a585f640 1 -- 172.21.15.100:0/4093556275 wait complete. 2024-03-29T11:46:08.149 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:08.144+0000 7fd1a585f640 1 librados: shutdown 2024-03-29T11:46:08.168 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:14.975 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:15.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.104+0000 7f3708857640 10 monclient: get_monmap_and_config 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.104+0000 7f3708857640 10 monclient: build_initial_monmap 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.105+0000 7f3708857640 10 monclient: monmap: 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:46:15.105994+0000 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:46:15.105994+0000 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:46:15.110 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:46:15.111 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:15.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.105+0000 7f3708857640 1 Processor -- start 2024-03-29T11:46:15.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.105+0000 7f3708857640 1 -- start start 2024-03-29T11:46:15.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.105+0000 7f3708857640 10 monclient: init 2024-03-29T11:46:15.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.106+0000 7f3708857640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:15.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.106+0000 7f3708857640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:46:15.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.106+0000 7f3708857640 10 monclient(hunting): picked mon.noname-b con 0x7f37041457f0 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:15.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.107+0000 7f3708857640 10 monclient(hunting): picked mon.noname-a con 0x7f370413f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.107+0000 7f3708857640 10 monclient(hunting): picked mon.noname-c con 0x7f37040a9d70 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.107+0000 7f3708857640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37040aaef0 con 0x7f370413f650 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.107+0000 7f3708857640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3704145bd0 con 0x7f37040a9d70 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.107+0000 7f3708857640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3704145eb0 con 0x7f37041457f0 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.107+0000 7f3708857640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.107+0000 7f3708857640 10 monclient(hunting): authenticate will time out at 1495.302743s 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.107+0000 7f3701d9b640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f370413f650 0x7f37040b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:35578/0 (socket says 172.21.15.100:35578) 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.107+0000 7f3701d9b640 1 -- 172.21.15.100:0/2457207580 learned_addr learned my addr 172.21.15.100:0/2457207580 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.108+0000 7f370159a640 1 -- 172.21.15.100:0/2457207580 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230379262 0 0) 0x7f37040aaef0 con 0x7f370413f650 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.108+0000 7f370159a640 10 monclient(hunting): _negotiate 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.108+0000 7f370159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.108+0000 7f370159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:15.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.108+0000 7f370159a640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:46:15.114 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.108+0000 7f370159a640 1 -- 172.21.15.100:0/2457207580 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ec003520 con 0x7f370413f650 2024-03-29T11:46:15.114 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.108+0000 7f370159a640 1 -- 172.21.15.100:0/2457207580 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362505694 0 0) 0x7f36ec003520 con 0x7f370413f650 2024-03-29T11:46:15.114 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.108+0000 7f370159a640 1 -- 172.21.15.100:0/2457207580 >> v1:172.21.15.100:6790/0 conn(0x7f37041457f0 legacy=0x7f370413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:46:15.114 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.108+0000 7f370159a640 1 -- 172.21.15.100:0/2457207580 >> v1:172.21.15.191:6789/0 conn(0x7f37040a9d70 legacy=0x7f370414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 10 monclient: _finish_hunting 0 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 1 monclient: found mon.noname-a 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 1 -- 172.21.15.100:0/2457207580 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3704146340 con 0x7f370413f650 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 10 monclient: _finish_auth 0 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 10 monclient: _check_auth_tickets 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 1 -- 172.21.15.100:0/2457207580 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f36f8001690 con 0x7f370413f650 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:15.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.109+0000 7f370159a640 10 monclient: dump: 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.110+0000 7f370159a640 1 -- 172.21.15.100:0/2457207580 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36f8001860 con 0x7f370413f650 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.110+0000 7f370159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.110+0000 7f370159a640 1 -- 172.21.15.100:0/2457207580 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f36f8001cb0 con 0x7f370413f650 2024-03-29T11:46:15.116 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.110+0000 7f370159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.110+0000 7f370159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.110+0000 7f370159a640 10 monclient: dump: 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.110+0000 7f3708857640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:46:15.117 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.110+0000 7f3708857640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:15.118 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.110+0000 7f3708857640 4 set_mon_vals no callback set 2024-03-29T11:46:15.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.111+0000 7f3708857640 10 monclient: shutdown 2024-03-29T11:46:15.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.111+0000 7f3708857640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:15.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.111+0000 7f3708857640 1 -- 172.21.15.100:0/2457207580 >> v1:172.21.15.100:6789/0 conn(0x7f370413f650 legacy=0x7f37040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:15.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.111+0000 7f3708857640 1 -- 172.21.15.100:0/2457207580 shutdown_connections 2024-03-29T11:46:15.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.111+0000 7f3708857640 1 -- 172.21.15.100:0/2457207580 >> 172.21.15.100:0/2457207580 conn(0x7f3704066a00 msgr2=0x7f3704066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.111+0000 7f3708857640 1 -- 172.21.15.100:0/2457207580 shutdown_connections 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.111+0000 7f3708857640 1 -- 172.21.15.100:0/2457207580 wait complete. 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.113+0000 7f3708857640 10 monclient: build_initial_monmap 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.113+0000 7f3708857640 10 monclient: monmap: 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:46:15.126 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.113+0000 7f3708857640 1 librados: starting msgr at 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.113+0000 7f3708857640 1 librados: starting objecter 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.113+0000 7f3708857640 1 Processor -- start 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.113+0000 7f3708857640 1 -- start start 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.113+0000 7f3708857640 1 librados: setting wanted keys 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.113+0000 7f3708857640 1 librados: calling monclient init 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.113+0000 7f3708857640 10 monclient: init 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.114+0000 7f3708857640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.114+0000 7f3708857640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.114+0000 7f3708857640 10 monclient(hunting): picked mon.noname-a con 0x7f37040a9d70 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.114+0000 7f370259c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f37040a9d70 0x7f3704135630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:36832/0 (socket says 172.21.15.100:36832) 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.114+0000 7f3708857640 10 monclient(hunting): picked mon.noname-c con 0x7f370413f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:15.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.114+0000 7f3708857640 10 monclient(hunting): picked mon.noname-b con 0x7f37041457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.114+0000 7f3708857640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3704070680 con 0x7f37040a9d70 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.114+0000 7f3708857640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3704070160 con 0x7f37041457f0 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f3708857640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f370406f3d0 con 0x7f370413f650 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f3708857640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f3708857640 10 monclient(hunting): authenticate will time out at 1495.310231s 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f370259c640 1 -- 172.21.15.100:0/1829437144 learned_addr learned my addr 172.21.15.100:0/1829437144 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 client.?.objecter ms_handle_connect 0x7f37040a9d70 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 client.?.objecter ms_handle_connect 0x7f370413f650 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370069115 0 0) 0x7f36f4001010 con 0x7f37040a9d70 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:15.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e4004eb0 con 0x7f37040a9d70 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602160869 0 0) 0x7f370406f3d0 con 0x7f370413f650 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.115+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36f4001010 con 0x7f370413f650 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1844292990 0 0) 0x7f36e4004eb0 con 0x7f37040a9d70 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f370406f3d0 con 0x7f37040a9d70 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 299988515 0 0) 0x7f36f4001010 con 0x7f370413f650 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e4004eb0 con 0x7f370413f650 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f36f40017c0 con 0x7f37040a9d70 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:15.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f36f8001b70 con 0x7f370413f650 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:15.130 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2763897356 0 0) 0x7f36e4004eb0 con 0x7f370413f650 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.116+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 >> v1:172.21.15.191:6789/0 conn(0x7f37041457f0 legacy=0x7f37041e6cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:46:15.131 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.117+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 >> v1:172.21.15.100:6789/0 conn(0x7f37040a9d70 legacy=0x7f3704135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.117+0000 7f36f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.117+0000 7f36f2ffd640 1 monclient: found mon.c 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.117+0000 7f36f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.117+0000 7f36f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.117+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3704071ec0 con 0x7f370413f650 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.117+0000 7f36f2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.117+0000 7f36f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.117+0000 7f36f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.118+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36f8001010 con 0x7f370413f650 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.118+0000 7f36f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.118+0000 7f3700d99640 4 set_mon_vals no callback set 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f36f8004bf0 con 0x7f370413f650 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f36f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f36f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:15.132 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f36f2ffd640 10 monclient: dump: 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f3708857640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f3708857640 10 monclient: _renew_subs 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f3708857640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f3708857640 1 -- 172.21.15.100:0/1829437144 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3704135d40 con 0x7f370413f650 2024-03-29T11:46:15.133 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f3708857640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:15.134 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f3708857640 10 monclient: _renew_subs 2024-03-29T11:46:15.134 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f3708857640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:15.134 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.119+0000 7f3708857640 1 -- 172.21.15.100:0/1829437144 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f370406fe20 con 0x7f370413f650 2024-03-29T11:46:15.134 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.120+0000 7f3708857640 1 librados: init done 2024-03-29T11:46:15.134 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.122+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f36f801abc0 con 0x7f370413f650 2024-03-29T11:46:15.134 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f370259c640 10 monclient: get_auth_request con 0x7f36e406de60 auth_method 0 2024-03-29T11:46:15.135 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7f36f8084950 con 0x7f370413f650 2024-03-29T11:46:15.135 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f36f2ffd640 10 client.4661.objecter ms_dispatch 0x7f3704000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:15.135 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f36f2ffd640 3 client.4661.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:15.135 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f36f2ffd640 3 client.4661.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:15.135 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f36f2ffd640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T11:46:15.135 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f36f2ffd640 10 client.4661.objecter ms_handle_connect 0x7f36e406de60 2024-03-29T11:46:15.135 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f3708857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:15.135 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f3708857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:15.135 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f3708857640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:15.135 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.123+0000 7f3708857640 1 -- 172.21.15.100:0/1829437144 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3704071870 con 0x7f370413f650 2024-03-29T11:46:15.136 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.131+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36f8050230 con 0x7f370413f650 2024-03-29T11:46:15.136 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.131+0000 7f36f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:15.136 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.131+0000 7f36f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:15.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.455+0000 7f3708857640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:15.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.455+0000 7f3708857640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:15.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.455+0000 7f3708857640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:15.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.455+0000 7f3708857640 1 -- 172.21.15.100:0/1829437144 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3704075f10 con 0x7f370413f650 2024-03-29T11:46:15.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.456+0000 7f36f2ffd640 1 -- 172.21.15.100:0/1829437144 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7f36f8052160 con 0x7f370413f650 2024-03-29T11:46:15.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.456+0000 7f36f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:15.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.456+0000 7f36f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:15.462 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:46:15.462 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:15.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.460+0000 7f3708857640 10 librados: watch_flush enter 2024-03-29T11:46:15.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.460+0000 7f3708857640 10 librados: watch_flush exit 2024-03-29T11:46:15.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.460+0000 7f3708857640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T11:46:15.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.460+0000 7f3708857640 10 client.4661.objecter successfully canceled tick 2024-03-29T11:46:15.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.460+0000 7f3708857640 1 -- 172.21.15.100:0/1829437144 >> v1:172.21.15.100:6816/3220289657 conn(0x7f36e406de60 legacy=0x7f36e4070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:15.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.460+0000 7f3708857640 10 monclient: shutdown 2024-03-29T11:46:15.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.460+0000 7f3708857640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:15.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.461+0000 7f3708857640 1 -- 172.21.15.100:0/1829437144 >> v1:172.21.15.100:6790/0 conn(0x7f370413f650 legacy=0x7f37041db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:15.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.463+0000 7f3708857640 1 -- 172.21.15.100:0/1829437144 shutdown_connections 2024-03-29T11:46:15.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.463+0000 7f3708857640 1 -- 172.21.15.100:0/1829437144 >> 172.21.15.100:0/1829437144 conn(0x7f3704066a00 msgr2=0x7f37040b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:15.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.464+0000 7f3708857640 1 -- 172.21.15.100:0/1829437144 shutdown_connections 2024-03-29T11:46:15.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.464+0000 7f3708857640 1 -- 172.21.15.100:0/1829437144 wait complete. 2024-03-29T11:46:15.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:15.464+0000 7f3708857640 1 librados: shutdown 2024-03-29T11:46:15.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:22.294 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:22.440 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.435+0000 7ff8818c9640 10 monclient: get_monmap_and_config 2024-03-29T11:46:22.440 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.435+0000 7ff8818c9640 10 monclient: build_initial_monmap 2024-03-29T11:46:22.440 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.435+0000 7ff8818c9640 10 monclient: monmap: 2024-03-29T11:46:22.440 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:22.440 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:22.440 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:46:22.436327+0000 2024-03-29T11:46:22.441 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:46:22.436327+0000 2024-03-29T11:46:22.441 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:22.441 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:22.441 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:22.441 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:46:22.441 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:46:22.441 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:22.442 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.438+0000 7ff8818c9640 1 Processor -- start 2024-03-29T11:46:22.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.438+0000 7ff8818c9640 1 -- start start 2024-03-29T11:46:22.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.438+0000 7ff8818c9640 10 monclient: init 2024-03-29T11:46:22.448 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.439+0000 7ff8818c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.439+0000 7ff8818c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.439+0000 7ff8818c9640 10 monclient(hunting): picked mon.noname-a con 0x7ff874038c60 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.439+0000 7ff8818c9640 10 monclient(hunting): picked mon.noname-b con 0x7ff8740324b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.439+0000 7ff8818c9640 10 monclient(hunting): picked mon.noname-c con 0x7ff8740d11e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.439+0000 7ff8818c9640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8740caf80 con 0x7ff874038c60 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.439+0000 7ff8818c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff874039850 con 0x7ff8740d11e0 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.439+0000 7ff8818c9640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8740c78d0 con 0x7ff8740324b0 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.439+0000 7ff8818c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.439+0000 7ff8818c9640 10 monclient(hunting): authenticate will time out at 1502.634298s 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.442+0000 7ff8808c7640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7ff874038c60 0x7ff8740c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:40179/0 (socket says 172.21.15.100:40179) 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.442+0000 7ff8808c7640 1 -- 172.21.15.100:0/1062913319 learned_addr learned my addr 172.21.15.100:0/1062913319 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551303796 0 0) 0x7ff8740caf80 con 0x7ff874038c60 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:46:22.449 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:22.450 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:22.450 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:46:22.451 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff86c003520 con 0x7ff874038c60 2024-03-29T11:46:22.451 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53211454 0 0) 0x7ff8740c78d0 con 0x7ff8740324b0 2024-03-29T11:46:22.451 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:46:22.451 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:22.451 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:22.451 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:46:22.451 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8740caf80 con 0x7ff8740324b0 2024-03-29T11:46:22.451 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3647394404 0 0) 0x7ff86c003520 con 0x7ff874038c60 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 >> v1:172.21.15.100:6790/0 conn(0x7ff8740324b0 legacy=0x7ff8740cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 >> v1:172.21.15.191:6789/0 conn(0x7ff8740d11e0 legacy=0x7ff8740d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8740c7a60 con 0x7ff874038c60 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff87c04f730 con 0x7ff874038c60 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:22.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: dump: 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff8818c9640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff8818c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:22.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff87c075f10 con 0x7ff874038c60 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 1 -- 172.21.15.100:0/1062913319 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff87c0758b0 con 0x7ff874038c60 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff87b7fe640 10 monclient: dump: 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:22.454 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff8818c9640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.443+0000 7ff8818c9640 4 set_mon_vals no callback set 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.444+0000 7ff8818c9640 10 monclient: shutdown 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.444+0000 7ff8818c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.444+0000 7ff8818c9640 1 -- 172.21.15.100:0/1062913319 >> v1:172.21.15.100:6789/0 conn(0x7ff874038c60 legacy=0x7ff8740c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.445+0000 7ff8818c9640 1 -- 172.21.15.100:0/1062913319 shutdown_connections 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.445+0000 7ff8818c9640 1 -- 172.21.15.100:0/1062913319 >> 172.21.15.100:0/1062913319 conn(0x7ff874005c20 msgr2=0x7ff874006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.445+0000 7ff8818c9640 1 -- 172.21.15.100:0/1062913319 shutdown_connections 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.445+0000 7ff8818c9640 1 -- 172.21.15.100:0/1062913319 wait complete. 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.446+0000 7ff8818c9640 10 monclient: build_initial_monmap 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.446+0000 7ff8818c9640 10 monclient: monmap: 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:22.455 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.446+0000 7ff8818c9640 1 librados: starting msgr at 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.446+0000 7ff8818c9640 1 librados: starting objecter 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.446+0000 7ff8818c9640 1 Processor -- start 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.446+0000 7ff8818c9640 1 -- start start 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.446+0000 7ff8818c9640 1 librados: setting wanted keys 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.446+0000 7ff8818c9640 1 librados: calling monclient init 2024-03-29T11:46:22.456 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.446+0000 7ff8818c9640 10 monclient: init 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.447+0000 7ff8818c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.447+0000 7ff8818c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.447+0000 7ff8818c9640 10 monclient(hunting): picked mon.noname-c con 0x7ff8740324b0 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.447+0000 7ff8808c7640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7ff8740324b0 0x7ff87402f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:35460/0 (socket says 172.21.15.100:35460) 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.450+0000 7ff8808c7640 1 -- 172.21.15.100:0/2866364974 learned_addr learned my addr 172.21.15.100:0/2866364974 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.450+0000 7ff8818c9640 10 monclient(hunting): picked mon.noname-a con 0x7ff874038c60 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.451+0000 7ff8797fa640 10 client.?.objecter ms_handle_connect 0x7ff8740324b0 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.451+0000 7ff8797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.451+0000 7ff8797fa640 10 client.?.objecter ms_handle_connect 0x7ff874038c60 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.451+0000 7ff8797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.451+0000 7ff8818c9640 10 monclient(hunting): picked mon.noname-b con 0x7ff8740d11e0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:22.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.451+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff874161ad0 con 0x7ff874038c60 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.451+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff874160bc0 con 0x7ff8740d11e0 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.451+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff874160ea0 con 0x7ff8740324b0 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.451+0000 7ff8818c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.451+0000 7ff8818c9640 10 monclient(hunting): authenticate will time out at 1502.646765s 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304408214 0 0) 0x7ff874161ad0 con 0x7ff874038c60 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): _negotiate 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff860004eb0 con 0x7ff874038c60 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314358749 0 0) 0x7ff874160ea0 con 0x7ff8740324b0 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): _negotiate 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:22.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:22.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:46:22.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff874161ad0 con 0x7ff8740324b0 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 490837766 0 0) 0x7ff860004eb0 con 0x7ff874038c60 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff874160ea0 con 0x7ff874038c60 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff870001930 con 0x7ff874038c60 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): dump: 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:22.461 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.453+0000 7ff8797fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.454+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894529762 0 0) 0x7ff874161ad0 con 0x7ff8740324b0 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.454+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff860004eb0 con 0x7ff8740324b0 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff87c04f400 con 0x7ff8740324b0 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient(hunting): dump: 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:22.462 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2360685765 0 0) 0x7ff860004eb0 con 0x7ff8740324b0 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 >> v1:172.21.15.191:6789/0 conn(0x7ff8740d11e0 legacy=0x7ff8741613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 >> v1:172.21.15.100:6789/0 conn(0x7ff874038c60 legacy=0x7ff8740348b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient: _finish_hunting 0 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 1 monclient: found mon.c 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:22.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87416f490 con 0x7ff8740324b0 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient: _finish_auth 0 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient: _check_auth_tickets 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8818c9640 5 monclient: authenticate success, global_id 4667 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8818c9640 10 monclient: _renew_subs 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8818c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87416c460 con 0x7ff8740324b0 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8818c9640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8818c9640 10 monclient: _renew_subs 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8818c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87416c8f0 con 0x7ff8740324b0 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8818c9640 1 librados: init done 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff87c04fdd0 con 0x7ff8740324b0 2024-03-29T11:46:22.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:22.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7ff87c067bf0 con 0x7ff8740324b0 2024-03-29T11:46:22.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:22.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:22.466 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff8797fa640 10 monclient: dump: 2024-03-29T11:46:22.466 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.455+0000 7ff87a7fc640 4 set_mon_vals no callback set 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.457+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7ff87c067d50 con 0x7ff8740324b0 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.458+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978172370 0 0) 0x7ff87c0ff470 con 0x7ff8740324b0 2024-03-29T11:46:22.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.458+0000 7ff8797fa640 10 client.4667.objecter ms_dispatch 0x7ff87c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:46:22.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.458+0000 7ff8797fa640 3 client.4667.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:46:22.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.458+0000 7ff8797fa640 3 client.4667.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:46:22.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.458+0000 7ff8818c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:22.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.458+0000 7ff8818c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:22.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.458+0000 7ff8818c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:22.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.458+0000 7ff8797fa640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-29T11:46:22.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.458+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85c005080 con 0x7ff8740324b0 2024-03-29T11:46:22.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.459+0000 7ff87bfff640 10 monclient: get_auth_request con 0x7ff86006dd60 auth_method 0 2024-03-29T11:46:22.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.459+0000 7ff8797fa640 10 client.4667.objecter ms_handle_connect 0x7ff86006dd60 2024-03-29T11:46:22.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.467+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff87c0cad50 con 0x7ff8740324b0 2024-03-29T11:46:22.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.467+0000 7ff8797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:22.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.467+0000 7ff8797fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:22.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.799+0000 7ff8818c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:22.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.799+0000 7ff8818c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:22.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.799+0000 7ff8818c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:22.804 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.799+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff85c005330 con 0x7ff8740324b0 2024-03-29T11:46:22.806 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.801+0000 7ff8797fa640 1 -- 172.21.15.100:0/2866364974 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2254504353) 0x7ff87c0ccc80 con 0x7ff8740324b0 2024-03-29T11:46:22.806 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.801+0000 7ff8797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:22.806 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.801+0000 7ff8797fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:22.807 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:46:22.807 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":10,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:36:26.144275+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:36:26.144275+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:22.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.805+0000 7ff8818c9640 10 librados: watch_flush enter 2024-03-29T11:46:22.810 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.805+0000 7ff8818c9640 10 librados: watch_flush exit 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.805+0000 7ff8818c9640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.805+0000 7ff8818c9640 10 client.4667.objecter successfully canceled tick 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.806+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 >> v1:172.21.15.100:6816/3220289657 conn(0x7ff86006dd60 legacy=0x7ff860070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.806+0000 7ff8818c9640 10 monclient: shutdown 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.806+0000 7ff8818c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.806+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 >> v1:172.21.15.100:6790/0 conn(0x7ff8740324b0 legacy=0x7ff87402f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.806+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 shutdown_connections 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.806+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 >> 172.21.15.100:0/2866364974 conn(0x7ff874005c20 msgr2=0x7ff87403ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.806+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 shutdown_connections 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.806+0000 7ff8818c9640 1 -- 172.21.15.100:0/2866364974 wait complete. 2024-03-29T11:46:22.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:22.806+0000 7ff8818c9640 1 librados: shutdown 2024-03-29T11:46:22.828 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:29.636 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:29.770 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.765+0000 7fe5267c8640 10 monclient: get_monmap_and_config 2024-03-29T11:46:29.774 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.765+0000 7fe5267c8640 10 monclient: build_initial_monmap 2024-03-29T11:46:29.774 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.765+0000 7fe5267c8640 10 monclient: monmap: 2024-03-29T11:46:29.774 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:29.774 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:29.774 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:46:29.766630+0000 2024-03-29T11:46:29.774 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:46:29.766630+0000 2024-03-29T11:46:29.775 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:29.775 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:29.775 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:29.775 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:46:29.775 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:46:29.775 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:29.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.766+0000 7fe5267c8640 1 Processor -- start 2024-03-29T11:46:29.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.766+0000 7fe5267c8640 1 -- start start 2024-03-29T11:46:29.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.766+0000 7fe5267c8640 10 monclient: init 2024-03-29T11:46:29.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.768+0000 7fe5267c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:29.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.768+0000 7fe5267c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:46:29.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.768+0000 7fe5267c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe5201457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:29.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.768+0000 7fe5267c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe52013f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:29.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.768+0000 7fe5267c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe5200a9d70 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:29.777 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.768+0000 7fe51f7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fe52013f650 0x7fe5200b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:57860/0 (socket says 172.21.15.100:57860) 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.768+0000 7fe51f7fe640 1 -- 172.21.15.100:0/95934207 learned_addr learned my addr 172.21.15.100:0/95934207 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe5267c8640 1 -- 172.21.15.100:0/95934207 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5200aaef0 con 0x7fe5200a9d70 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe5267c8640 1 -- 172.21.15.100:0/95934207 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe520145bd0 con 0x7fe5201457f0 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe5267c8640 1 -- 172.21.15.100:0/95934207 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe520145eb0 con 0x7fe52013f650 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe5267c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe5267c8640 10 monclient(hunting): authenticate will time out at 1509.964511s 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe51effd640 1 -- 172.21.15.100:0/95934207 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951129489 0 0) 0x7fe520145eb0 con 0x7fe52013f650 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe51effd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe51effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe51effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe51effd640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe51effd640 1 -- 172.21.15.100:0/95934207 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe508003520 con 0x7fe52013f650 2024-03-29T11:46:29.778 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe51effd640 1 -- 172.21.15.100:0/95934207 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1372938485 0 0) 0x7fe508003520 con 0x7fe52013f650 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe51effd640 1 -- 172.21.15.100:0/95934207 >> v1:172.21.15.191:6789/0 conn(0x7fe5201457f0 legacy=0x7fe52013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.769+0000 7fe51effd640 1 -- 172.21.15.100:0/95934207 >> v1:172.21.15.100:6789/0 conn(0x7fe5200a9d70 legacy=0x7fe52014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: _finish_hunting 0 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 1 monclient: found mon.noname-b 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 1 -- 172.21.15.100:0/95934207 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe520146340 con 0x7fe52013f650 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: _finish_auth 0 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: _check_auth_tickets 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 1 -- 172.21.15.100:0/95934207 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe510001690 con 0x7fe52013f650 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: dump: 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:29.779 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe5267c8640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe5267c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 1 -- 172.21.15.100:0/95934207 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe510001920 con 0x7fe52013f650 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 1 -- 172.21.15.100:0/95934207 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe510001d30 con 0x7fe52013f650 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:29.780 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.771+0000 7fe51effd640 10 monclient: dump: 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.772+0000 7fe5267c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:29.781 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.772+0000 7fe5267c8640 4 set_mon_vals no callback set 2024-03-29T11:46:29.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.773+0000 7fe5267c8640 10 monclient: shutdown 2024-03-29T11:46:29.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.773+0000 7fe5267c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:29.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.773+0000 7fe5267c8640 1 -- 172.21.15.100:0/95934207 >> v1:172.21.15.100:6790/0 conn(0x7fe52013f650 legacy=0x7fe5200b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:29.783 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.773+0000 7fe5267c8640 1 -- 172.21.15.100:0/95934207 shutdown_connections 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.773+0000 7fe5267c8640 1 -- 172.21.15.100:0/95934207 >> 172.21.15.100:0/95934207 conn(0x7fe520066a00 msgr2=0x7fe520066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.773+0000 7fe5267c8640 1 -- 172.21.15.100:0/95934207 shutdown_connections 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.773+0000 7fe5267c8640 1 -- 172.21.15.100:0/95934207 wait complete. 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.774+0000 7fe5267c8640 10 monclient: build_initial_monmap 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.774+0000 7fe5267c8640 10 monclient: monmap: 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:46:29.784 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.775+0000 7fe5267c8640 1 librados: starting msgr at 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.775+0000 7fe5267c8640 1 librados: starting objecter 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.775+0000 7fe5267c8640 1 Processor -- start 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.775+0000 7fe5267c8640 1 -- start start 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.775+0000 7fe5267c8640 1 librados: setting wanted keys 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.775+0000 7fe5267c8640 1 librados: calling monclient init 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.775+0000 7fe5267c8640 10 monclient: init 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.777+0000 7fe5267c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.777+0000 7fe5267c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.777+0000 7fe51ffff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fe5200a9d70 0x7fe520141fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:59854/0 (socket says 172.21.15.100:59854) 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.777+0000 7fe51ffff640 1 -- 172.21.15.100:0/1742222349 learned_addr learned my addr 172.21.15.100:0/1742222349 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.777+0000 7fe5267c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe5200a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.777+0000 7fe51cff9640 10 client.?.objecter ms_handle_connect 0x7fe5200a9d70 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.777+0000 7fe51cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:29.785 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe5267c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe52013f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe5267c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe5201457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe520142810 con 0x7fe52013f650 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe520141b70 con 0x7fe5201457f0 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe520141d00 con 0x7fe5200a9d70 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe5267c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe5267c8640 10 monclient(hunting): authenticate will time out at 1509.973658s 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179251274 0 0) 0x7fe520141d00 con 0x7fe5200a9d70 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe51cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe51cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe51cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe51cff9640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:46:29.786 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.778+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe50c004eb0 con 0x7fe5200a9d70 2024-03-29T11:46:29.791 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.779+0000 7fe51cff9640 10 client.?.objecter ms_handle_connect 0x7fe5201457f0 2024-03-29T11:46:29.791 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.779+0000 7fe51cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:29.791 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.779+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1890713663 0 0) 0x7fe50c004eb0 con 0x7fe5200a9d70 2024-03-29T11:46:29.791 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.779+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe520141d00 con 0x7fe5200a9d70 2024-03-29T11:46:29.791 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.779+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe514001990 con 0x7fe5200a9d70 2024-03-29T11:46:29.791 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.779+0000 7fe51cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:29.791 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.779+0000 7fe51cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:46:29.791 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.779+0000 7fe51cff9640 10 monclient(hunting): dump: 2024-03-29T11:46:29.791 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.779+0000 7fe51cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.779+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964983548 0 0) 0x7fe520141b70 con 0x7fe5201457f0 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:29.792 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe50c004eb0 con 0x7fe5201457f0 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 10 client.?.objecter ms_handle_connect 0x7fe52013f650 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2738601897 0 0) 0x7fe520141d00 con 0x7fe5200a9d70 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 >> v1:172.21.15.191:6789/0 conn(0x7fe5201457f0 legacy=0x7fe5201da550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 >> v1:172.21.15.100:6789/0 conn(0x7fe52013f650 legacy=0x7fe5200b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 1 monclient: found mon.c 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5201445c0 con 0x7fe5200a9d70 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 10 monclient: _finish_auth 0 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe51cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:29.793 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe5267c8640 5 monclient: authenticate success, global_id 4673 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe5267c8640 10 monclient: _renew_subs 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe5267c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe520144810 con 0x7fe5200a9d70 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe5267c8640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe5267c8640 10 monclient: _renew_subs 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe5267c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5200b1330 con 0x7fe5200a9d70 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.780+0000 7fe5267c8640 1 librados: init done 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.781+0000 7fe5267c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.781+0000 7fe5267c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.781+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe514001c50 con 0x7fe5200a9d70 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.781+0000 7fe5267c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.781+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe520075f10 con 0x7fe5200a9d70 2024-03-29T11:46:29.794 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.781+0000 7fe51cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.781+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fe514003a60 con 0x7fe5200a9d70 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.781+0000 7fe51cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.781+0000 7fe51cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.781+0000 7fe51cff9640 10 monclient: dump: 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:29.795 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.782+0000 7fe51e7fc640 4 set_mon_vals no callback set 2024-03-29T11:46:29.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.782+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fe514003c80 con 0x7fe5200a9d70 2024-03-29T11:46:29.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.783+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1127350977 0 0) 0x7fe514084a40 con 0x7fe5200a9d70 2024-03-29T11:46:29.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.783+0000 7fe51cff9640 10 client.4673.objecter ms_dispatch 0x7fe520000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:46:29.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.783+0000 7fe51cff9640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:46:29.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.783+0000 7fe51cff9640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:46:29.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.783+0000 7fe51cff9640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T11:46:29.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.783+0000 7fe51f7fe640 10 monclient: get_auth_request con 0x7fe50c06dd90 auth_method 0 2024-03-29T11:46:29.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.784+0000 7fe51cff9640 10 client.4673.objecter ms_handle_connect 0x7fe50c06dd90 2024-03-29T11:46:29.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.789+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe514050320 con 0x7fe5200a9d70 2024-03-29T11:46:29.796 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.789+0000 7fe51cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:29.797 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:29.789+0000 7fe51cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:30.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.110+0000 7fe5267c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:30.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.110+0000 7fe5267c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:30.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.110+0000 7fe5267c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:30.115 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.110+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5200b3680 con 0x7fe5200a9d70 2024-03-29T11:46:30.116 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.111+0000 7fe51cff9640 1 -- 172.21.15.100:0/1742222349 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1916490066) 0x7fe514052250 con 0x7fe5200a9d70 2024-03-29T11:46:30.116 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.111+0000 7fe51cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:30.116 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.111+0000 7fe51cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:30.119 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:46:30.119 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":11,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:46:28.911584+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:46:28.911584+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:30.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.117+0000 7fe5267c8640 10 librados: watch_flush enter 2024-03-29T11:46:30.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.117+0000 7fe5267c8640 10 librados: watch_flush exit 2024-03-29T11:46:30.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.117+0000 7fe5267c8640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T11:46:30.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.117+0000 7fe5267c8640 10 client.4673.objecter successfully canceled tick 2024-03-29T11:46:30.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.117+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 >> v1:172.21.15.100:6816/3220289657 conn(0x7fe50c06dd90 legacy=0x7fe50c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:30.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.117+0000 7fe5267c8640 10 monclient: shutdown 2024-03-29T11:46:30.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.117+0000 7fe5267c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:30.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.117+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 >> v1:172.21.15.100:6790/0 conn(0x7fe5200a9d70 legacy=0x7fe520141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:30.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.118+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 shutdown_connections 2024-03-29T11:46:30.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.118+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 >> 172.21.15.100:0/1742222349 conn(0x7fe520066a00 msgr2=0x7fe520066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:30.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.119+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 shutdown_connections 2024-03-29T11:46:30.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.119+0000 7fe5267c8640 1 -- 172.21.15.100:0/1742222349 wait complete. 2024-03-29T11:46:30.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:30.119+0000 7fe5267c8640 1 librados: shutdown 2024-03-29T11:46:30.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:36.946 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:37.094 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.089+0000 7f845d0ae640 10 monclient: get_monmap_and_config 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.089+0000 7f845d0ae640 10 monclient: build_initial_monmap 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.089+0000 7f845d0ae640 10 monclient: monmap: 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:46:37.090838+0000 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:46:37.090838+0000 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:46:37.095 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:46:37.096 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:37.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.091+0000 7f845d0ae640 1 Processor -- start 2024-03-29T11:46:37.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.091+0000 7f845d0ae640 1 -- start start 2024-03-29T11:46:37.096 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.091+0000 7f845d0ae640 10 monclient: init 2024-03-29T11:46:37.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.092+0000 7f845d0ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:37.097 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.092+0000 7f845d0ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:46:37.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.092+0000 7f845d0ae640 10 monclient(hunting): picked mon.noname-a con 0x7f8458071de0 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:37.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.092+0000 7f845d0ae640 10 monclient(hunting): picked mon.noname-c con 0x7f84580721c0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:37.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.092+0000 7f845d0ae640 10 monclient(hunting): picked mon.noname-b con 0x7f84581425d0 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:37.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.092+0000 7f845d0ae640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845813fb10 con 0x7f8458071de0 2024-03-29T11:46:37.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.092+0000 7f845d0ae640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8458142c20 con 0x7f84580721c0 2024-03-29T11:46:37.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.092+0000 7f845d0ae640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84580ada50 con 0x7f84581425d0 2024-03-29T11:46:37.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.092+0000 7f845d0ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:37.098 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.092+0000 7f845d0ae640 10 monclient(hunting): authenticate will time out at 1517.287743s 2024-03-29T11:46:37.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.093+0000 7f845759e640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f84581425d0 0x7f845814da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:54012/0 (socket says 172.21.15.100:54012) 2024-03-29T11:46:37.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.093+0000 7f845759e640 1 -- 172.21.15.100:0/2936997427 learned_addr learned my addr 172.21.15.100:0/2936997427 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:37.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.093+0000 7f8455d9b640 1 -- 172.21.15.100:0/2936997427 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769793083 0 0) 0x7f84580ada50 con 0x7f84581425d0 2024-03-29T11:46:37.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.093+0000 7f8455d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:46:37.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.093+0000 7f8455d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:37.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.093+0000 7f8455d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:37.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.093+0000 7f8455d9b640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:46:37.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.093+0000 7f8455d9b640 1 -- 172.21.15.100:0/2936997427 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8440003520 con 0x7f84581425d0 2024-03-29T11:46:37.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.094+0000 7f8455d9b640 1 -- 172.21.15.100:0/2936997427 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2408491680 0 0) 0x7f8440003520 con 0x7f84581425d0 2024-03-29T11:46:37.099 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.094+0000 7f8455d9b640 1 -- 172.21.15.100:0/2936997427 >> v1:172.21.15.191:6789/0 conn(0x7f84580721c0 legacy=0x7f8458145670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:46:37.100 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.095+0000 7f8455d9b640 1 -- 172.21.15.100:0/2936997427 >> v1:172.21.15.100:6789/0 conn(0x7f8458071de0 legacy=0x7f845813f400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:46:37.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:46:37.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 1 monclient: found mon.noname-b 2024-03-29T11:46:37.102 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:37.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:46:37.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 1 -- 172.21.15.100:0/2936997427 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84580b0500 con 0x7f84581425d0 2024-03-29T11:46:37.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 10 monclient: _finish_auth 0 2024-03-29T11:46:37.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:46:37.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:37.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 1 -- 172.21.15.100:0/2936997427 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8448001690 con 0x7f84581425d0 2024-03-29T11:46:37.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:37.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:46:37.105 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f8455d9b640 10 monclient: dump: 2024-03-29T11:46:37.105 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f845d0ae640 5 monclient: authenticate success, global_id 4676 2024-03-29T11:46:37.106 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.097+0000 7f845d0ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.098+0000 7f8455d9b640 1 -- 172.21.15.100:0/2936997427 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84480018f0 con 0x7f84581425d0 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.098+0000 7f8455d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.098+0000 7f8455d9b640 1 -- 172.21.15.100:0/2936997427 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8448001d40 con 0x7f84581425d0 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.098+0000 7f8455d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.098+0000 7f8455d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.098+0000 7f8455d9b640 10 monclient: dump: 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:37.107 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:37.108 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:37.108 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:37.108 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:37.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.098+0000 7f845d0ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:37.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.098+0000 7f845d0ae640 4 set_mon_vals no callback set 2024-03-29T11:46:37.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.099+0000 7f845d0ae640 10 monclient: shutdown 2024-03-29T11:46:37.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.099+0000 7f845d0ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:37.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.099+0000 7f845d0ae640 1 -- 172.21.15.100:0/2936997427 >> v1:172.21.15.100:6790/0 conn(0x7f84581425d0 legacy=0x7f845814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:37.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.099+0000 7f845d0ae640 1 -- 172.21.15.100:0/2936997427 shutdown_connections 2024-03-29T11:46:37.108 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.099+0000 7f845d0ae640 1 -- 172.21.15.100:0/2936997427 >> 172.21.15.100:0/2936997427 conn(0x7f845806b660 msgr2=0x7f845806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:37.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.100+0000 7f845d0ae640 1 -- 172.21.15.100:0/2936997427 shutdown_connections 2024-03-29T11:46:37.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.101+0000 7f845d0ae640 1 -- 172.21.15.100:0/2936997427 wait complete. 2024-03-29T11:46:37.109 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.102+0000 7f845d0ae640 10 monclient: build_initial_monmap 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.102+0000 7f845d0ae640 10 monclient: monmap: 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.102+0000 7f845d0ae640 1 librados: starting msgr at 2024-03-29T11:46:37.110 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.102+0000 7f845d0ae640 1 librados: starting objecter 2024-03-29T11:46:37.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 1 Processor -- start 2024-03-29T11:46:37.111 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 1 -- start start 2024-03-29T11:46:37.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 1 librados: setting wanted keys 2024-03-29T11:46:37.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 1 librados: calling monclient init 2024-03-29T11:46:37.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 10 monclient: init 2024-03-29T11:46:37.112 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 10 monclient(hunting): picked mon.noname-c con 0x7f8458071de0 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 10 monclient(hunting): picked mon.noname-a con 0x7f84581425d0 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 10 monclient(hunting): picked mon.noname-b con 0x7f8458135f50 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845813c970 con 0x7f84581425d0 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845813b740 con 0x7f8458135f50 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845813b070 con 0x7f8458071de0 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.103+0000 7f845d0ae640 10 monclient(hunting): authenticate will time out at 1517.299006s 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.105+0000 7f8456d9d640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f8458071de0 0x7f845813a860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:42598/0 (socket says 172.21.15.100:42598) 2024-03-29T11:46:37.113 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.106+0000 7f8456d9d640 1 -- 172.21.15.100:0/867378664 learned_addr learned my addr 172.21.15.100:0/867378664 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:37.114 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.108+0000 7f843f7fe640 10 client.?.objecter ms_handle_connect 0x7f8458071de0 2024-03-29T11:46:37.114 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.108+0000 7f843f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:37.114 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.109+0000 7f843f7fe640 10 client.?.objecter ms_handle_connect 0x7f8458135f50 2024-03-29T11:46:37.114 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.109+0000 7f843f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.109+0000 7f843f7fe640 10 client.?.objecter ms_handle_connect 0x7f84581425d0 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.109+0000 7f843f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188112704 0 0) 0x7f845813b070 con 0x7f8458071de0 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8444004eb0 con 0x7f8458071de0 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161785100 0 0) 0x7f845813c970 con 0x7f84581425d0 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:37.120 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845813b070 con 0x7f84581425d0 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2646310802 0 0) 0x7f8444004eb0 con 0x7f8458071de0 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f845813c970 con 0x7f8458071de0 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2314042541 0 0) 0x7f845813b070 con 0x7f84581425d0 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 --> v1:172.21.15.100:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8444004eb0 con 0x7f84581425d0 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f844c001960 con 0x7f8458071de0 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): dump: 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:37.121 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.110+0000 7f843f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8450001a90 con 0x7f84581425d0 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 10 monclient(hunting): dump: 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:37.122 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.0 v1:172.21.15.100:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1632852618 0 0) 0x7f8444004eb0 con 0x7f84581425d0 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 >> v1:172.21.15.100:6790/0 conn(0x7f8458071de0 legacy=0x7f845813a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 >> v1:172.21.15.191:6789/0 conn(0x7f8458135f50 legacy=0x7f84580b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:46:37.123 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 1 monclient: found mon.a 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845813b200 con 0x7f84581425d0 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f843f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f845d0ae640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f845d0ae640 10 monclient: _renew_subs 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f845d0ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f845d0ae640 1 -- 172.21.15.100:0/867378664 --> v1:172.21.15.100:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845813b450 con 0x7f84581425d0 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f845d0ae640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f845d0ae640 10 monclient: _renew_subs 2024-03-29T11:46:37.124 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f845d0ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f845d0ae640 1 -- 172.21.15.100:0/867378664 --> v1:172.21.15.100:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84580b2ff0 con 0x7f84581425d0 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.111+0000 7f845d0ae640 1 librados: init done 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.113+0000 7f845d0ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.113+0000 7f845d0ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.113+0000 7f845d0ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.113+0000 7f845d0ae640 1 -- 172.21.15.100:0/867378664 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845813b8d0 con 0x7f84581425d0 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.114+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.0 v1:172.21.15.100:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8450001bf0 con 0x7f84581425d0 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.114+0000 7f843f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.114+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.0 v1:172.21.15.100:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8450003a10 con 0x7f84581425d0 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.114+0000 7f843f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.114+0000 7f843f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.114+0000 7f843f7fe640 10 monclient: dump: 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:37.125 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:37.126 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:37.126 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:37.126 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:37.126 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:37.126 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:37.126 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:37.126 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:37.126 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:37.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.115+0000 7f845559a640 4 set_mon_vals no callback set 2024-03-29T11:46:37.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.116+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.0 v1:172.21.15.100:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f84500199e0 con 0x7f84581425d0 2024-03-29T11:46:37.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.117+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.0 v1:172.21.15.100:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1127350977 0 0) 0x7f84500838a0 con 0x7f84581425d0 2024-03-29T11:46:37.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.117+0000 7f843f7fe640 10 client.4554.objecter ms_dispatch 0x7f8458000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:46:37.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.117+0000 7f843f7fe640 3 client.4554.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:46:37.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.117+0000 7f843f7fe640 3 client.4554.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:46:37.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.117+0000 7f843f7fe640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:46:37.127 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.120+0000 7f8456d9d640 10 monclient: get_auth_request con 0x7f844406df90 auth_method 0 2024-03-29T11:46:37.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.123+0000 7f843f7fe640 10 client.4554.objecter ms_handle_connect 0x7f844406df90 2024-03-29T11:46:37.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.124+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.0 v1:172.21.15.100:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f845004f180 con 0x7f84581425d0 2024-03-29T11:46:37.128 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.124+0000 7f843f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:37.129 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.124+0000 7f843f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:37.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.452+0000 7f845d0ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:37.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.452+0000 7f845d0ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:37.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.452+0000 7f845d0ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.100:6789/0 2024-03-29T11:46:37.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.452+0000 7f845d0ae640 1 -- 172.21.15.100:0/867378664 --> v1:172.21.15.100:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8458075f10 con 0x7f84581425d0 2024-03-29T11:46:37.459 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.455+0000 7f843f7fe640 1 -- 172.21.15.100:0/867378664 <== mon.0 v1:172.21.15.100:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1916490066) 0x7f84500510b0 con 0x7f84581425d0 2024-03-29T11:46:37.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.455+0000 7f843f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:37.460 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.455+0000 7f843f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:37.460 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:46:37.460 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":11,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:46:28.911584+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:46:28.911584+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:37.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.458+0000 7f845d0ae640 10 librados: watch_flush enter 2024-03-29T11:46:37.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.459+0000 7f845d0ae640 10 librados: watch_flush exit 2024-03-29T11:46:37.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.459+0000 7f845d0ae640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:46:37.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.459+0000 7f845d0ae640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:46:37.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.459+0000 7f845d0ae640 1 -- 172.21.15.100:0/867378664 >> v1:172.21.15.100:6816/3220289657 conn(0x7f844406df90 legacy=0x7f8444070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:37.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.459+0000 7f845d0ae640 10 monclient: shutdown 2024-03-29T11:46:37.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.459+0000 7f845d0ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:37.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.459+0000 7f845d0ae640 1 -- 172.21.15.100:0/867378664 >> v1:172.21.15.100:6789/0 conn(0x7f84581425d0 legacy=0x7f8458135840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:37.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.460+0000 7f845d0ae640 1 -- 172.21.15.100:0/867378664 shutdown_connections 2024-03-29T11:46:37.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.460+0000 7f845d0ae640 1 -- 172.21.15.100:0/867378664 >> 172.21.15.100:0/867378664 conn(0x7f845806b660 msgr2=0x7f84580a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:37.467 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.463+0000 7f845d0ae640 1 -- 172.21.15.100:0/867378664 shutdown_connections 2024-03-29T11:46:37.468 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.463+0000 7f845d0ae640 1 -- 172.21.15.100:0/867378664 wait complete. 2024-03-29T11:46:37.469 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:37.464+0000 7f845d0ae640 1 librados: shutdown 2024-03-29T11:46:37.486 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:44.295 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:44.443 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.438+0000 7f1d0b0ac640 10 monclient: get_monmap_and_config 2024-03-29T11:46:44.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.438+0000 7f1d0b0ac640 10 monclient: build_initial_monmap 2024-03-29T11:46:44.444 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.438+0000 7f1d0b0ac640 10 monclient: monmap: 2024-03-29T11:46:44.444 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:44.444 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:44.444 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:46:44.439798+0000 2024-03-29T11:46:44.444 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:46:44.439798+0000 2024-03-29T11:46:44.444 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:44.444 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:44.444 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:44.444 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:46:44.445 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:46:44.445 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:44.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.439+0000 7f1d0b0ac640 1 Processor -- start 2024-03-29T11:46:44.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.439+0000 7f1d0b0ac640 1 -- start start 2024-03-29T11:46:44.445 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.439+0000 7f1d0b0ac640 10 monclient: init 2024-03-29T11:46:44.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.440+0000 7f1d0b0ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:44.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.440+0000 7f1d0b0ac640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:46:44.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.440+0000 7f1d0b0ac640 10 monclient(hunting): picked mon.noname-b con 0x7f1d040b2800 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:44.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.440+0000 7f1d0b0ac640 10 monclient(hunting): picked mon.noname-a con 0x7f1d040b3220 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:44.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.440+0000 7f1d0b0ac640 10 monclient(hunting): picked mon.noname-c con 0x7f1d041439c0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:44.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.440+0000 7f1d0b0ac640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d040b3840 con 0x7f1d040b3220 2024-03-29T11:46:44.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.441+0000 7f1d0b0ac640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0406f230 con 0x7f1d041439c0 2024-03-29T11:46:44.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.441+0000 7f1d0b0ac640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0406ef50 con 0x7f1d040b2800 2024-03-29T11:46:44.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.441+0000 7f1d0b0ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:44.447 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.441+0000 7f1d0b0ac640 10 monclient(hunting): authenticate will time out at 1524.636099s 2024-03-29T11:46:44.450 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.444+0000 7f1d0a0aa640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f1d040b2800 0x7f1d041432b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:41052/0 (socket says 172.21.15.100:41052) 2024-03-29T11:46:44.450 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.444+0000 7f1d0a0aa640 1 -- 172.21.15.100:0/1374007871 learned_addr learned my addr 172.21.15.100:0/1374007871 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:44.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.447+0000 7f1d090a8640 1 -- 172.21.15.100:0/1374007871 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364308312 0 0) 0x7f1d0406f230 con 0x7f1d041439c0 2024-03-29T11:46:44.452 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.447+0000 7f1d090a8640 10 monclient(hunting): _negotiate 2024-03-29T11:46:44.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.447+0000 7f1d090a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:44.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.447+0000 7f1d090a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:44.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.447+0000 7f1d090a8640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:46:44.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.447+0000 7f1d090a8640 1 -- 172.21.15.100:0/1374007871 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf0003520 con 0x7f1d041439c0 2024-03-29T11:46:44.453 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.447+0000 7f1d090a8640 1 -- 172.21.15.100:0/1374007871 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94597488 0 0) 0x7f1d040b3840 con 0x7f1d040b3220 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient(hunting): _negotiate 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 1 -- 172.21.15.100:0/1374007871 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d0406f230 con 0x7f1d040b3220 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 1 -- 172.21.15.100:0/1374007871 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4206111884 0 0) 0x7f1d0406f230 con 0x7f1d040b3220 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 1 -- 172.21.15.100:0/1374007871 >> v1:172.21.15.100:6790/0 conn(0x7f1d040b2800 legacy=0x7f1d041432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 1 -- 172.21.15.100:0/1374007871 >> v1:172.21.15.191:6789/0 conn(0x7f1d041439c0 legacy=0x7f1d04151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient: _finish_hunting 0 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 1 monclient: found mon.noname-a 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:44.454 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 1 -- 172.21.15.100:0/1374007871 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d04071d30 con 0x7f1d040b3220 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient: _finish_auth 0 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient: _check_auth_tickets 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 1 -- 172.21.15.100:0/1374007871 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1d00001670 con 0x7f1d040b3220 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient: dump: 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:44.455 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:44.456 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:44.456 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:44.456 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:44.456 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:44.456 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:44.456 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:44.456 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:44.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d0b0ac640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:46:44.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d0b0ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:44.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 1 -- 172.21.15.100:0/1374007871 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d000017d0 con 0x7f1d040b3220 2024-03-29T11:46:44.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d090a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:44.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d0b0ac640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:44.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.448+0000 7f1d0b0ac640 4 set_mon_vals no callback set 2024-03-29T11:46:44.457 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.449+0000 7f1d0b0ac640 10 monclient: shutdown 2024-03-29T11:46:44.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.449+0000 7f1d0b0ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:44.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.449+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/1374007871 >> v1:172.21.15.100:6789/0 conn(0x7f1d040b3220 legacy=0x7f1d04146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:44.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.452+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/1374007871 shutdown_connections 2024-03-29T11:46:44.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.452+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/1374007871 >> 172.21.15.100:0/1374007871 conn(0x7f1d04066a00 msgr2=0x7f1d04066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:44.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.452+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/1374007871 shutdown_connections 2024-03-29T11:46:44.458 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.452+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/1374007871 wait complete. 2024-03-29T11:46:44.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.453+0000 7f1d0b0ac640 10 monclient: build_initial_monmap 2024-03-29T11:46:44.462 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.453+0000 7f1d0b0ac640 10 monclient: monmap: 2024-03-29T11:46:44.462 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.453+0000 7f1d0b0ac640 1 librados: starting msgr at 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.453+0000 7f1d0b0ac640 1 librados: starting objecter 2024-03-29T11:46:44.463 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.453+0000 7f1d0b0ac640 1 Processor -- start 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 1 -- start start 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 1 librados: setting wanted keys 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 1 librados: calling monclient init 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 10 monclient: init 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 10 monclient(hunting): picked mon.noname-a con 0x7f1d040b2800 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 10 monclient(hunting): picked mon.noname-c con 0x7f1d040b3220 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 10 monclient(hunting): picked mon.noname-b con 0x7f1d041439c0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0406fb00 con 0x7f1d040b2800 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d040b1a60 con 0x7f1d041439c0 2024-03-29T11:46:44.464 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d040b6e00 con 0x7f1d040b3220 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.454+0000 7f1d0b0ac640 10 monclient(hunting): authenticate will time out at 1524.649877s 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.457+0000 7f1d0a8ab640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f1d041439c0 0x7f1d04070cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.100:42437/0 (socket says 172.21.15.100:42437) 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.457+0000 7f1d0a8ab640 1 -- 172.21.15.100:0/3337739361 learned_addr learned my addr 172.21.15.100:0/3337739361 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.457+0000 7f1cf6ffd640 10 client.?.objecter ms_handle_connect 0x7f1d041439c0 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.457+0000 7f1cf6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.457+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956117643 0 0) 0x7f1d040b1a60 con 0x7f1d041439c0 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.457+0000 7f1cf6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.457+0000 7f1cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.457+0000 7f1cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.457+0000 7f1cf6ffd640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:46:44.465 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.457+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce8004eb0 con 0x7f1d041439c0 2024-03-29T11:46:44.471 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.458+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2580860178 0 0) 0x7f1ce8004eb0 con 0x7f1d041439c0 2024-03-29T11:46:44.471 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.458+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d040b1a60 con 0x7f1d041439c0 2024-03-29T11:46:44.471 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.458+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1cfc001c30 con 0x7f1d041439c0 2024-03-29T11:46:44.471 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.458+0000 7f1cf6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:44.471 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.458+0000 7f1cf6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:46:44.471 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.458+0000 7f1cf6ffd640 10 monclient(hunting): dump: 2024-03-29T11:46:44.471 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:44.471 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:44.471 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.458+0000 7f1cf6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.458+0000 7f1cf6ffd640 10 client.?.objecter ms_handle_connect 0x7f1d040b3220 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.458+0000 7f1cf6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 10 client.?.objecter ms_handle_connect 0x7f1d040b2800 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:44.472 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341022155 0 0) 0x7f1d0406fb00 con 0x7f1d040b2800 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce8004eb0 con 0x7f1d040b2800 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1293567611 0 0) 0x7f1d040b1a60 con 0x7f1d041439c0 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 >> v1:172.21.15.100:6790/0 conn(0x7f1d040b3220 legacy=0x7f1d040705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 >> v1:172.21.15.100:6789/0 conn(0x7f1d040b2800 legacy=0x7f1d041e41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 1 monclient: found mon.b 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:44.473 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d040b6f90 con 0x7f1d041439c0 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1cf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1d0b0ac640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1d0b0ac640 10 monclient: _renew_subs 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1d0b0ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/3337739361 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d040b1560 con 0x7f1d041439c0 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1d0b0ac640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1d0b0ac640 10 monclient: _renew_subs 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1d0b0ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:46:44.474 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/3337739361 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d040b6a20 con 0x7f1d041439c0 2024-03-29T11:46:44.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.461+0000 7f1d0b0ac640 1 librados: init done 2024-03-29T11:46:44.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.462+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cfc003990 con 0x7f1d041439c0 2024-03-29T11:46:44.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.462+0000 7f1cf6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:44.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.462+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f1cfc003de0 con 0x7f1d041439c0 2024-03-29T11:46:44.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.462+0000 7f1cf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:44.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.462+0000 7f1cf6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:46:44.475 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.462+0000 7f1cf6ffd640 10 monclient: dump: 2024-03-29T11:46:44.475 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:44.475 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.462+0000 7f1cf7fff640 4 set_mon_vals no callback set 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.463+0000 7f1d0b0ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.463+0000 7f1d0b0ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:44.476 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.463+0000 7f1d0b0ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:46:44.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.463+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/3337739361 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec005080 con 0x7f1d041439c0 2024-03-29T11:46:44.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.464+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f1cfc003f40 con 0x7f1d041439c0 2024-03-29T11:46:44.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.465+0000 7f1d0a0aa640 10 monclient: get_auth_request con 0x7f1ce806dc90 auth_method 0 2024-03-29T11:46:44.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.465+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1127350977 0 0) 0x7f1cfc084d50 con 0x7f1d041439c0 2024-03-29T11:46:44.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.465+0000 7f1cf6ffd640 10 client.4519.objecter ms_dispatch 0x7f1d04000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:46:44.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.465+0000 7f1cf6ffd640 3 client.4519.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:46:44.477 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.465+0000 7f1cf6ffd640 3 client.4519.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:46:44.478 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.465+0000 7f1cf6ffd640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-29T11:46:44.478 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.465+0000 7f1cf6ffd640 10 client.4519.objecter ms_handle_connect 0x7f1ce806dc90 2024-03-29T11:46:44.479 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.474+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1cfc050630 con 0x7f1d041439c0 2024-03-29T11:46:44.479 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.474+0000 7f1cf6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:44.480 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.474+0000 7f1cf6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:44.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.800+0000 7f1d0b0ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:44.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.800+0000 7f1d0b0ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:44.805 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.800+0000 7f1d0b0ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:46:44.806 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.800+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/3337739361 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cec005330 con 0x7f1d041439c0 2024-03-29T11:46:44.808 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:46:44.808 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":11,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:46:28.911584+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:46:28.911584+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:44.808 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.803+0000 7f1cf6ffd640 1 -- 172.21.15.100:0/3337739361 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1916490066) 0x7f1cfc052560 con 0x7f1d041439c0 2024-03-29T11:46:44.808 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.803+0000 7f1cf6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:44.808 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.803+0000 7f1cf6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:44.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.806+0000 7f1d0b0ac640 10 librados: watch_flush enter 2024-03-29T11:46:44.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.806+0000 7f1d0b0ac640 10 librados: watch_flush exit 2024-03-29T11:46:44.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.806+0000 7f1d0b0ac640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-29T11:46:44.811 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.806+0000 7f1d0b0ac640 10 client.4519.objecter successfully canceled tick 2024-03-29T11:46:44.812 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.807+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/3337739361 >> v1:172.21.15.100:6816/3220289657 conn(0x7f1ce806dc90 legacy=0x7f1ce8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:44.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.807+0000 7f1d0b0ac640 10 monclient: shutdown 2024-03-29T11:46:44.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.807+0000 7f1d0b0ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:44.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.807+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/3337739361 >> v1:172.21.15.191:6789/0 conn(0x7f1d041439c0 legacy=0x7f1d04070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:44.813 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.808+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/3337739361 shutdown_connections 2024-03-29T11:46:44.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.808+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/3337739361 >> 172.21.15.100:0/3337739361 conn(0x7f1d04066a00 msgr2=0x7f1d04069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:44.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.808+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/3337739361 shutdown_connections 2024-03-29T11:46:44.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.808+0000 7f1d0b0ac640 1 -- 172.21.15.100:0/3337739361 wait complete. 2024-03-29T11:46:44.814 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:44.809+0000 7f1d0b0ac640 1 librados: shutdown 2024-03-29T11:46:44.833 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:51.641 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:51.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.814+0000 7f3fa7596640 10 monclient: get_monmap_and_config 2024-03-29T11:46:51.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.814+0000 7f3fa7596640 10 monclient: build_initial_monmap 2024-03-29T11:46:51.819 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.814+0000 7f3fa7596640 10 monclient: monmap: 2024-03-29T11:46:51.819 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:51.819 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:51.819 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:46:51.815117+0000 2024-03-29T11:46:51.819 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:46:51.815117+0000 2024-03-29T11:46:51.819 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:51.819 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:51.820 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:51.820 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:46:51.820 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:46:51.820 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:51.820 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.814+0000 7f3fa7596640 1 Processor -- start 2024-03-29T11:46:51.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.815+0000 7f3fa7596640 1 -- start start 2024-03-29T11:46:51.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.815+0000 7f3fa7596640 10 monclient: init 2024-03-29T11:46:51.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.816+0000 7f3fa7596640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:51.824 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.816+0000 7f3fa7596640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:46:51.825 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.816+0000 7f3fa7596640 10 monclient(hunting): picked mon.noname-a con 0x7f3fa00b0770 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:51.825 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.816+0000 7f3fa7596640 10 monclient(hunting): picked mon.noname-b con 0x7f3fa014a700 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:51.825 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.816+0000 7f3fa7596640 10 monclient(hunting): picked mon.noname-c con 0x7f3fa00ac5a0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:51.825 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.816+0000 7f3fa7596640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa00aba70 con 0x7f3fa00b0770 2024-03-29T11:46:51.825 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.816+0000 7f3fa7596640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa00aecb0 con 0x7f3fa00ac5a0 2024-03-29T11:46:51.825 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.816+0000 7f3fa7596640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa00af150 con 0x7f3fa014a700 2024-03-29T11:46:51.825 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.816+0000 7f3fa7596640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:51.825 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.816+0000 7f3fa7596640 10 monclient(hunting): authenticate will time out at 1532.011246s 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.818+0000 7f3fa4b31640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f3fa014a700 0x7f3fa00abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:34820/0 (socket says 172.21.15.100:34820) 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.818+0000 7f3fa4b31640 1 -- 172.21.15.100:0/673870675 learned_addr learned my addr 172.21.15.100:0/673870675 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.819+0000 7f3f97fff640 1 -- 172.21.15.100:0/673870675 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040125819 0 0) 0x7f3fa00af150 con 0x7f3fa014a700 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.819+0000 7f3f97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.819+0000 7f3f97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.819+0000 7f3f97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.819+0000 7f3f97fff640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.819+0000 7f3f97fff640 1 -- 172.21.15.100:0/673870675 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f8c003520 con 0x7f3fa014a700 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.820+0000 7f3f97fff640 1 -- 172.21.15.100:0/673870675 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628833962 0 0) 0x7f3f8c003520 con 0x7f3fa014a700 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.820+0000 7f3f97fff640 1 -- 172.21.15.100:0/673870675 >> v1:172.21.15.191:6789/0 conn(0x7f3fa00ac5a0 legacy=0x7f3fa0151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 1 -- 172.21.15.100:0/673870675 >> v1:172.21.15.100:6789/0 conn(0x7f3fa00b0770 legacy=0x7f3fa0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:51.827 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 10 monclient: _finish_hunting 0 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 1 monclient: found mon.noname-b 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 1 -- 172.21.15.100:0/673870675 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa00ac9d0 con 0x7f3fa014a700 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 10 monclient: _finish_auth 0 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 10 monclient: _check_auth_tickets 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3fa7596640 5 monclient: authenticate success, global_id 4685 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3fa7596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 1 -- 172.21.15.100:0/673870675 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3f9c001690 con 0x7f3fa014a700 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 10 monclient: dump: 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:51.828 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3fa7596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 1 -- 172.21.15.100:0/673870675 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f9c001920 con 0x7f3fa014a700 2024-03-29T11:46:51.829 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3f97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:51.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3fa7596640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:51.830 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.821+0000 7f3fa7596640 4 set_mon_vals no callback set 2024-03-29T11:46:51.831 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.822+0000 7f3fa7596640 10 monclient: shutdown 2024-03-29T11:46:51.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.822+0000 7f3fa7596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:51.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.822+0000 7f3fa7596640 1 -- 172.21.15.100:0/673870675 >> v1:172.21.15.100:6790/0 conn(0x7f3fa014a700 legacy=0x7f3fa00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:51.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.822+0000 7f3fa7596640 1 -- 172.21.15.100:0/673870675 shutdown_connections 2024-03-29T11:46:51.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.822+0000 7f3fa7596640 1 -- 172.21.15.100:0/673870675 >> 172.21.15.100:0/673870675 conn(0x7f3fa006b660 msgr2=0x7f3fa006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:51.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.823+0000 7f3fa7596640 1 -- 172.21.15.100:0/673870675 shutdown_connections 2024-03-29T11:46:51.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.823+0000 7f3fa7596640 1 -- 172.21.15.100:0/673870675 wait complete. 2024-03-29T11:46:51.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.824+0000 7f3fa7596640 10 monclient: build_initial_monmap 2024-03-29T11:46:51.832 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.824+0000 7f3fa7596640 10 monclient: monmap: 2024-03-29T11:46:51.832 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:51.832 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.825+0000 7f3fa7596640 1 librados: starting msgr at 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.825+0000 7f3fa7596640 1 librados: starting objecter 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.825+0000 7f3fa7596640 1 Processor -- start 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.825+0000 7f3fa7596640 1 -- start start 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.825+0000 7f3fa7596640 1 librados: setting wanted keys 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.825+0000 7f3fa7596640 1 librados: calling monclient init 2024-03-29T11:46:51.833 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.825+0000 7f3fa7596640 10 monclient: init 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.826+0000 7f3fa7596640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.826+0000 7f3fa7596640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.826+0000 7f3fa7596640 10 monclient(hunting): picked mon.noname-b con 0x7f3fa00ac5a0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.828+0000 7f3fa7596640 10 monclient(hunting): picked mon.noname-c con 0x7f3fa00b0770 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.829+0000 7f3fa4b31640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f3fa00b0770 0x7f3fa01446e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:38608/0 (socket says 172.21.15.100:38608) 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.829+0000 7f3fa7596640 10 monclient(hunting): picked mon.noname-a con 0x7f3fa014a700 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.829+0000 7f3fa7596640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa0145500 con 0x7f3fa014a700 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.829+0000 7f3fa7596640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa0125910 con 0x7f3fa00ac5a0 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.830+0000 7f3fa4b31640 1 -- 172.21.15.100:0/3604101058 learned_addr learned my addr 172.21.15.100:0/3604101058 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.830+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa0126110 con 0x7f3fa00b0770 2024-03-29T11:46:51.835 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.830+0000 7f3fa7596640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:51.836 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.830+0000 7f3fa7596640 10 monclient(hunting): authenticate will time out at 1532.025086s 2024-03-29T11:46:51.836 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.830+0000 7f3f95ffb640 10 client.?.objecter ms_handle_connect 0x7f3fa00b0770 2024-03-29T11:46:51.836 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.830+0000 7f3f95ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:51.837 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.831+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369750055 0 0) 0x7f3fa0126110 con 0x7f3fa00b0770 2024-03-29T11:46:51.837 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.831+0000 7f3f95ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:46:51.837 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.831+0000 7f3f95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:51.837 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.831+0000 7f3f95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:51.837 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.831+0000 7f3f95ffb640 10 monclient(hunting): my global_id is 4688 2024-03-29T11:46:51.837 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.831+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f84004eb0 con 0x7f3fa00b0770 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.831+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 66546054 0 0) 0x7f3f84004eb0 con 0x7f3fa00b0770 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.831+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fa0126110 con 0x7f3fa00b0770 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.832+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3f9c004980 con 0x7f3fa00b0770 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.832+0000 7f3f95ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.832+0000 7f3f95ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.832+0000 7f3f95ffb640 10 monclient(hunting): dump: 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:51.838 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:51.839 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:51.839 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:51.839 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:51.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.832+0000 7f3f95ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:51.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.832+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 204391375 0 0) 0x7f3fa0126110 con 0x7f3fa00b0770 2024-03-29T11:46:51.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.832+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 >> v1:172.21.15.191:6789/0 conn(0x7f3fa00ac5a0 legacy=0x7f3fa01de7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:46:51.839 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.832+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 >> v1:172.21.15.100:6789/0 conn(0x7f3fa014a700 legacy=0x7f3fa0144df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:46:51.841 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.836+0000 7f3f95ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:46:51.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.836+0000 7f3f95ffb640 1 monclient: found mon.c 2024-03-29T11:46:51.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.836+0000 7f3f95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:51.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.836+0000 7f3f95ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:51.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.836+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa0145720 con 0x7f3fa00b0770 2024-03-29T11:46:51.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.836+0000 7f3f95ffb640 10 monclient: _finish_auth 0 2024-03-29T11:46:51.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.836+0000 7f3f95ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:46:51.842 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.836+0000 7f3f95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:51.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.837+0000 7f3fa7596640 5 monclient: authenticate success, global_id 4688 2024-03-29T11:46:51.845 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.837+0000 7f3fa7596640 10 monclient: _renew_subs 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.837+0000 7f3fa7596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.837+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa0145970 con 0x7f3fa00b0770 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.837+0000 7f3fa7596640 10 client.4688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.837+0000 7f3fa7596640 10 monclient: _renew_subs 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.837+0000 7f3fa7596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.837+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa00705d0 con 0x7f3fa00b0770 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.837+0000 7f3fa7596640 1 librados: init done 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.838+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f9c001b10 con 0x7f3fa00b0770 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.838+0000 7f3f95ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.838+0000 7f3f977fe640 4 set_mon_vals no callback set 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.839+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f3f9c004fb0 con 0x7f3fa00b0770 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.839+0000 7f3f95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:51.846 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.839+0000 7f3f95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.839+0000 7f3f95ffb640 10 monclient: dump: 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.840+0000 7f3fa7596640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:51.847 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.840+0000 7f3fa7596640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:51.848 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.840+0000 7f3fa7596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:51.848 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.840+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f80005080 con 0x7f3fa00b0770 2024-03-29T11:46:51.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.843+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f3f9c005110 con 0x7f3fa00b0770 2024-03-29T11:46:51.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.843+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1127350977 0 0) 0x7f3f9c084d20 con 0x7f3fa00b0770 2024-03-29T11:46:51.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.843+0000 7f3f95ffb640 10 client.4688.objecter ms_dispatch 0x7f3fa0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:46:51.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.843+0000 7f3f95ffb640 3 client.4688.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:46:51.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.843+0000 7f3f95ffb640 3 client.4688.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:46:51.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.843+0000 7f3f95ffb640 20 client.4688.objecter dump_active .. 0 homeless 2024-03-29T11:46:51.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.843+0000 7f3fa5332640 10 monclient: get_auth_request con 0x7f3f8406dc20 auth_method 0 2024-03-29T11:46:51.849 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.844+0000 7f3f95ffb640 10 client.4688.objecter ms_handle_connect 0x7f3f8406dc20 2024-03-29T11:46:51.857 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.852+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f9c050600 con 0x7f3fa00b0770 2024-03-29T11:46:51.857 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.852+0000 7f3f95ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:51.857 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:51.852+0000 7f3f95ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:52.210 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.205+0000 7f3fa7596640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:52.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.205+0000 7f3fa7596640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:52.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.205+0000 7f3fa7596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:52.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.205+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f80005330 con 0x7f3fa00b0770 2024-03-29T11:46:52.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.207+0000 7f3f95ffb640 1 -- 172.21.15.100:0/3604101058 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1916490066) 0x7f3f9c052530 con 0x7f3fa00b0770 2024-03-29T11:46:52.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.207+0000 7f3f95ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:52.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.207+0000 7f3f95ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:52.213 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:46:52.213 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":11,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:46:28.911584+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:46:28.911584+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:52.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.211+0000 7f3fa7596640 10 librados: watch_flush enter 2024-03-29T11:46:52.216 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.211+0000 7f3fa7596640 10 librados: watch_flush exit 2024-03-29T11:46:52.217 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.211+0000 7f3fa7596640 20 client.4688.objecter shutdown clearing up homeless session... 2024-03-29T11:46:52.217 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.211+0000 7f3fa7596640 10 client.4688.objecter successfully canceled tick 2024-03-29T11:46:52.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.211+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 >> v1:172.21.15.100:6816/3220289657 conn(0x7f3f8406dc20 legacy=0x7f3f840700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:52.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.212+0000 7f3fa7596640 10 monclient: shutdown 2024-03-29T11:46:52.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.212+0000 7f3fa7596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:52.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.212+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 >> v1:172.21.15.100:6790/0 conn(0x7f3fa00b0770 legacy=0x7f3fa01446e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:52.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.212+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 shutdown_connections 2024-03-29T11:46:52.218 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.212+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 >> 172.21.15.100:0/3604101058 conn(0x7f3fa006b660 msgr2=0x7f3fa00b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:52.220 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.214+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 shutdown_connections 2024-03-29T11:46:52.220 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.215+0000 7f3fa7596640 1 -- 172.21.15.100:0/3604101058 wait complete. 2024-03-29T11:46:52.220 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:52.215+0000 7f3fa7596640 1 librados: shutdown 2024-03-29T11:46:52.239 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:59.052 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.182+0000 7f40f5c4d640 10 monclient: get_monmap_and_config 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.182+0000 7f40f5c4d640 10 monclient: build_initial_monmap 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.182+0000 7f40f5c4d640 10 monclient: monmap: 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:46:59.183632+0000 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:46:59.183632+0000 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:59.188 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:46:59.189 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:46:59.189 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:59.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.183+0000 7f40f5c4d640 1 Processor -- start 2024-03-29T11:46:59.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.183+0000 7f40f5c4d640 1 -- start start 2024-03-29T11:46:59.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.183+0000 7f40f5c4d640 10 monclient: init 2024-03-29T11:46:59.189 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.184+0000 7f40f5c4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:59.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.184+0000 7f40f5c4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:46:59.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.184+0000 7f40f5c4d640 10 monclient(hunting): picked mon.noname-c con 0x7f40f01457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:59.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.184+0000 7f40f5c4d640 10 monclient(hunting): picked mon.noname-b con 0x7f40f013f650 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:59.190 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.184+0000 7f40f5c4d640 10 monclient(hunting): picked mon.noname-a con 0x7f40f00a9d70 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:59.195 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40effff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f40f00a9d70 0x7f40f014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:51680/0 (socket says 172.21.15.100:51680) 2024-03-29T11:46:59.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40f5c4d640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f00aaef0 con 0x7f40f00a9d70 2024-03-29T11:46:59.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40effff640 1 -- 172.21.15.100:0/1382842263 learned_addr learned my addr 172.21.15.100:0/1382842263 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:59.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40f5c4d640 1 -- 172.21.15.100:0/1382842263 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f0145bd0 con 0x7f40f01457f0 2024-03-29T11:46:59.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40f5c4d640 1 -- 172.21.15.100:0/1382842263 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f0145eb0 con 0x7f40f013f650 2024-03-29T11:46:59.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40f5c4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:59.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40f5c4d640 10 monclient(hunting): authenticate will time out at 1539.380155s 2024-03-29T11:46:59.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40ee7fc640 1 -- 172.21.15.100:0/1382842263 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309096758 0 0) 0x7f40f0145eb0 con 0x7f40f013f650 2024-03-29T11:46:59.196 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:46:59.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:59.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:59.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40ee7fc640 10 monclient(hunting): my global_id is 4691 2024-03-29T11:46:59.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40ee7fc640 1 -- 172.21.15.100:0/1382842263 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d8003520 con 0x7f40f013f650 2024-03-29T11:46:59.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.185+0000 7f40ee7fc640 1 -- 172.21.15.100:0/1382842263 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3739862045 0 0) 0x7f40d8003520 con 0x7f40f013f650 2024-03-29T11:46:59.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.186+0000 7f40ee7fc640 1 -- 172.21.15.100:0/1382842263 >> v1:172.21.15.191:6789/0 conn(0x7f40f01457f0 legacy=0x7f40f013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:46:59.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 1 -- 172.21.15.100:0/1382842263 >> v1:172.21.15.100:6789/0 conn(0x7f40f00a9d70 legacy=0x7f40f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:59.197 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 1 monclient: found mon.noname-b 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 1 -- 172.21.15.100:0/1382842263 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f0146340 con 0x7f40f013f650 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 1 -- 172.21.15.100:0/1382842263 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f40e4001690 con 0x7f40f013f650 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: dump: 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:59.198 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40f5c4d640 5 monclient: authenticate success, global_id 4691 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40f5c4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 1 -- 172.21.15.100:0/1382842263 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40e4001860 con 0x7f40f013f650 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40f5c4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:59.199 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40f5c4d640 4 set_mon_vals no callback set 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 1 -- 172.21.15.100:0/1382842263 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f40e4001cb0 con 0x7f40f013f650 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.187+0000 7f40ee7fc640 10 monclient: dump: 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:59.200 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.189+0000 7f40f5c4d640 10 monclient: shutdown 2024-03-29T11:46:59.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.189+0000 7f40f5c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:59.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.189+0000 7f40f5c4d640 1 -- 172.21.15.100:0/1382842263 >> v1:172.21.15.100:6790/0 conn(0x7f40f013f650 legacy=0x7f40f00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:59.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.189+0000 7f40f5c4d640 1 -- 172.21.15.100:0/1382842263 shutdown_connections 2024-03-29T11:46:59.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.189+0000 7f40f5c4d640 1 -- 172.21.15.100:0/1382842263 >> 172.21.15.100:0/1382842263 conn(0x7f40f0066a00 msgr2=0x7f40f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:59.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.190+0000 7f40f5c4d640 1 -- 172.21.15.100:0/1382842263 shutdown_connections 2024-03-29T11:46:59.201 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.190+0000 7f40f5c4d640 1 -- 172.21.15.100:0/1382842263 wait complete. 2024-03-29T11:46:59.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.191+0000 7f40f5c4d640 10 monclient: build_initial_monmap 2024-03-29T11:46:59.204 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.191+0000 7f40f5c4d640 10 monclient: monmap: 2024-03-29T11:46:59.204 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:46:59.204 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:59.204 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.191+0000 7f40f5c4d640 1 librados: starting msgr at 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.191+0000 7f40f5c4d640 1 librados: starting objecter 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.191+0000 7f40f5c4d640 1 Processor -- start 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.192+0000 7f40f5c4d640 1 -- start start 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.192+0000 7f40f5c4d640 1 librados: setting wanted keys 2024-03-29T11:46:59.205 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.192+0000 7f40f5c4d640 1 librados: calling monclient init 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.192+0000 7f40f5c4d640 10 monclient: init 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40f5c4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40f5c4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40f5c4d640 10 monclient(hunting): picked mon.noname-c con 0x7f40f00a9d70 addr v1:172.21.15.100:6790/0 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40f5c4d640 10 monclient(hunting): picked mon.noname-a con 0x7f40f013f650 addr v1:172.21.15.100:6789/0 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40f5c4d640 10 monclient(hunting): picked mon.noname-b con 0x7f40f01457f0 addr v1:172.21.15.191:6789/0 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40f5c4d640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f01dc470 con 0x7f40f013f650 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40f5c4d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f0070540 con 0x7f40f01457f0 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40f5c4d640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f006f650 con 0x7f40f00a9d70 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40f5c4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40f5c4d640 10 monclient(hunting): authenticate will time out at 1539.388574s 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40ef7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f40f00a9d70 0x7f40f006fe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:60428/0 (socket says 172.21.15.100:60428) 2024-03-29T11:46:59.206 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40ef7fe640 1 -- 172.21.15.100:0/360376183 learned_addr learned my addr 172.21.15.100:0/360376183 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40d7fff640 10 client.?.objecter ms_handle_connect 0x7f40f00a9d70 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.193+0000 7f40d7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.194+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529833443 0 0) 0x7f40f006f650 con 0x7f40f00a9d70 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.194+0000 7f40d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.194+0000 7f40d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.194+0000 7f40d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.194+0000 7f40d7fff640 10 monclient(hunting): my global_id is 4694 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.194+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40dc004eb0 con 0x7f40f00a9d70 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.194+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2566684424 0 0) 0x7f40dc004eb0 con 0x7f40f00a9d70 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.194+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40f006f650 con 0x7f40f00a9d70 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.195+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f40e00017d0 con 0x7f40f00a9d70 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.195+0000 7f40d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.195+0000 7f40d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:46:59.207 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.195+0000 7f40d7fff640 10 monclient(hunting): dump: 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.195+0000 7f40d7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.195+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 908234032 0 0) 0x7f40f006f650 con 0x7f40f00a9d70 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.195+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 >> v1:172.21.15.191:6789/0 conn(0x7f40f01457f0 legacy=0x7f40f00b0670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:46:59.208 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 >> v1:172.21.15.100:6789/0 conn(0x7f40f013f650 legacy=0x7f40f006ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 1 monclient: found mon.c 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f00b17e0 con 0x7f40f00a9d70 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 10 monclient: _finish_auth 0 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 10 client.?.objecter ms_handle_connect 0x7f40f013f650 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.196+0000 7f40d7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40e0004860 con 0x7f40f00a9d70 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f40e00017d0 con 0x7f40f00a9d70 2024-03-29T11:46:59.209 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40d7fff640 10 monclient: dump: 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40f5c4d640 5 monclient: authenticate success, global_id 4694 2024-03-29T11:46:59.210 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40f5c4d640 10 monclient: _renew_subs 2024-03-29T11:46:59.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40f5c4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:59.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40f5c4d640 1 -- 172.21.15.100:0/360376183 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f00b0d80 con 0x7f40f00a9d70 2024-03-29T11:46:59.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40f5c4d640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:59.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40f5c4d640 10 monclient: _renew_subs 2024-03-29T11:46:59.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40f5c4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:59.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40f5c4d640 1 -- 172.21.15.100:0/360376183 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f0071f10 con 0x7f40f00a9d70 2024-03-29T11:46:59.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40f5c4d640 1 librados: init done 2024-03-29T11:46:59.211 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.197+0000 7f40edffb640 4 set_mon_vals no callback set 2024-03-29T11:46:59.212 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.200+0000 7f40f5c4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:59.212 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.200+0000 7f40f5c4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:59.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.200+0000 7f40f5c4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:59.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.200+0000 7f40f5c4d640 1 -- 172.21.15.100:0/360376183 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d0005080 con 0x7f40f00a9d70 2024-03-29T11:46:59.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.200+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f40e001a840 con 0x7f40f00a9d70 2024-03-29T11:46:59.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.201+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1127350977 0 0) 0x7f40e0054590 con 0x7f40f00a9d70 2024-03-29T11:46:59.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.201+0000 7f40d7fff640 10 client.4694.objecter ms_dispatch 0x7f40f0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:46:59.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.201+0000 7f40d7fff640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:46:59.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.201+0000 7f40d7fff640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:46:59.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.201+0000 7f40eeffd640 10 monclient: get_auth_request con 0x7f40dc06de70 auth_method 0 2024-03-29T11:46:59.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.201+0000 7f40d7fff640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-29T11:46:59.213 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.201+0000 7f40d7fff640 10 client.4694.objecter ms_handle_connect 0x7f40dc06de70 2024-03-29T11:46:59.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.209+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40e0050830 con 0x7f40f00a9d70 2024-03-29T11:46:59.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.209+0000 7f40d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:59.214 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.209+0000 7f40d7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:59.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.524+0000 7f40f5c4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:59.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.524+0000 7f40f5c4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:59.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.524+0000 7f40f5c4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:46:59.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.524+0000 7f40f5c4d640 1 -- 172.21.15.100:0/360376183 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40d0005330 con 0x7f40f00a9d70 2024-03-29T11:46:59.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.524+0000 7f40d7fff640 1 -- 172.21.15.100:0/360376183 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1916490066) 0x7f40e00509d0 con 0x7f40f00a9d70 2024-03-29T11:46:59.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.524+0000 7f40d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:59.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.524+0000 7f40d7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:59.531 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:46:59.531 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":11,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:46:28.911584+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:46:28.911584+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:59.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.528+0000 7f40f5c4d640 10 librados: watch_flush enter 2024-03-29T11:46:59.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.528+0000 7f40f5c4d640 10 librados: watch_flush exit 2024-03-29T11:46:59.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.528+0000 7f40f5c4d640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-29T11:46:59.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.528+0000 7f40f5c4d640 10 client.4694.objecter successfully canceled tick 2024-03-29T11:46:59.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.528+0000 7f40f5c4d640 1 -- 172.21.15.100:0/360376183 >> v1:172.21.15.100:6816/3220289657 conn(0x7f40dc06de70 legacy=0x7f40dc070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:59.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.528+0000 7f40f5c4d640 10 monclient: shutdown 2024-03-29T11:46:59.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.528+0000 7f40f5c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:59.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.528+0000 7f40f5c4d640 1 -- 172.21.15.100:0/360376183 >> v1:172.21.15.100:6790/0 conn(0x7f40f00a9d70 legacy=0x7f40f006fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:59.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.528+0000 7f40f5c4d640 1 -- 172.21.15.100:0/360376183 shutdown_connections 2024-03-29T11:46:59.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.528+0000 7f40f5c4d640 1 -- 172.21.15.100:0/360376183 >> 172.21.15.100:0/360376183 conn(0x7f40f0066a00 msgr2=0x7f40f0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:59.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.529+0000 7f40f5c4d640 1 -- 172.21.15.100:0/360376183 shutdown_connections 2024-03-29T11:46:59.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.529+0000 7f40f5c4d640 1 -- 172.21.15.100:0/360376183 wait complete. 2024-03-29T11:46:59.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:46:59.529+0000 7f40f5c4d640 1 librados: shutdown 2024-03-29T11:46:59.552 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:06.359 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:06.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.499+0000 7f60fccb3640 10 monclient: get_monmap_and_config 2024-03-29T11:47:06.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.499+0000 7f60fccb3640 10 monclient: build_initial_monmap 2024-03-29T11:47:06.505 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.499+0000 7f60fccb3640 10 monclient: monmap: 2024-03-29T11:47:06.505 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:47:06.505 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:06.505 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:47:06.500669+0000 2024-03-29T11:47:06.506 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:47:06.500669+0000 2024-03-29T11:47:06.506 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:06.506 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:06.506 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:47:06.506 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:47:06.506 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:47:06.506 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:06.506 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.500+0000 7f60fccb3640 1 Processor -- start 2024-03-29T11:47:06.510 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.500+0000 7f60fccb3640 1 -- start start 2024-03-29T11:47:06.510 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.500+0000 7f60fccb3640 10 monclient: init 2024-03-29T11:47:06.510 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60fccb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:06.510 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60fccb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60fccb3640 10 monclient(hunting): picked mon.noname-a con 0x7f60f80b0770 addr v1:172.21.15.100:6789/0 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60fccb3640 10 monclient(hunting): picked mon.noname-c con 0x7f60f814a700 addr v1:172.21.15.191:6789/0 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60fccb3640 10 monclient(hunting): picked mon.noname-b con 0x7f60f80ac5a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60fccb3640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f80aba70 con 0x7f60f80b0770 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60fccb3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f80aecb0 con 0x7f60f814a700 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60fccb3640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f80af150 con 0x7f60f80ac5a0 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60fccb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60fccb3640 10 monclient(hunting): authenticate will time out at 1546.696804s 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60f659c640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f60f80b0770 0x7f60f8149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:44826/0 (socket says 172.21.15.100:44826) 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.501+0000 7f60f659c640 1 -- 172.21.15.100:0/1728936712 learned_addr learned my addr 172.21.15.100:0/1728936712 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612460637 0 0) 0x7f60f80af150 con 0x7f60f80ac5a0 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): _negotiate 2024-03-29T11:47:06.511 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:06.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:06.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): my global_id is 4697 2024-03-29T11:47:06.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e0003520 con 0x7f60f80ac5a0 2024-03-29T11:47:06.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088294960 0 0) 0x7f60f80aecb0 con 0x7f60f814a700 2024-03-29T11:47:06.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): _negotiate 2024-03-29T11:47:06.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:06.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:06.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:47:06.512 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f80af150 con 0x7f60f814a700 2024-03-29T11:47:06.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813850 0 0) 0x7f60f80aba70 con 0x7f60f80b0770 2024-03-29T11:47:06.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): _negotiate 2024-03-29T11:47:06.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:06.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:06.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:47:06.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.502+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e0004a30 con 0x7f60f80b0770 2024-03-29T11:47:06.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 484271204 0 0) 0x7f60f80af150 con 0x7f60f814a700 2024-03-29T11:47:06.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 >> v1:172.21.15.100:6790/0 conn(0x7f60f80ac5a0 legacy=0x7f60f8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:06.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 >> v1:172.21.15.100:6789/0 conn(0x7f60f80b0770 legacy=0x7f60f8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:06.513 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: _finish_hunting 0 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 1 monclient: found mon.noname-c 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f80ac9d0 con 0x7f60f814a700 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: _finish_auth 0 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: _check_auth_tickets 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f60ec001520 con 0x7f60f814a700 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: dump: 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:06.514 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60fccb3640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60fccb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60ec001700 con 0x7f60f814a700 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60fccb3640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60fccb3640 4 set_mon_vals no callback set 2024-03-29T11:47:06.515 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 1 -- 172.21.15.100:0/1728936712 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f60ec001520 con 0x7f60f814a700 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.503+0000 7f60f559a640 10 monclient: dump: 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:06.516 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.504+0000 7f60fccb3640 10 monclient: shutdown 2024-03-29T11:47:06.517 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.504+0000 7f60fccb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:06.517 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.504+0000 7f60fccb3640 1 -- 172.21.15.100:0/1728936712 >> v1:172.21.15.191:6789/0 conn(0x7f60f814a700 legacy=0x7f60f80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:06.517 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.505+0000 7f60fccb3640 1 -- 172.21.15.100:0/1728936712 shutdown_connections 2024-03-29T11:47:06.517 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.505+0000 7f60fccb3640 1 -- 172.21.15.100:0/1728936712 >> 172.21.15.100:0/1728936712 conn(0x7f60f806b660 msgr2=0x7f60f806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:06.517 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.505+0000 7f60fccb3640 1 -- 172.21.15.100:0/1728936712 shutdown_connections 2024-03-29T11:47:06.517 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.505+0000 7f60fccb3640 1 -- 172.21.15.100:0/1728936712 wait complete. 2024-03-29T11:47:06.524 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.506+0000 7f60fccb3640 10 monclient: build_initial_monmap 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.506+0000 7f60fccb3640 10 monclient: monmap: 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:06.525 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.507+0000 7f60fccb3640 1 librados: starting msgr at 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.507+0000 7f60fccb3640 1 librados: starting objecter 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.507+0000 7f60fccb3640 1 Processor -- start 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.507+0000 7f60fccb3640 1 -- start start 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.507+0000 7f60fccb3640 1 librados: setting wanted keys 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.507+0000 7f60fccb3640 1 librados: calling monclient init 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.507+0000 7f60fccb3640 10 monclient: init 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.508+0000 7f60fccb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.508+0000 7f60fccb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.508+0000 7f60fccb3640 10 monclient(hunting): picked mon.noname-c con 0x7f60f80ac5a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.508+0000 7f60fccb3640 10 monclient(hunting): picked mon.noname-b con 0x7f60f80b0770 addr v1:172.21.15.191:6789/0 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60fccb3640 10 monclient(hunting): picked mon.noname-a con 0x7f60f814a700 addr v1:172.21.15.100:6789/0 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60f659c640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f60f80ac5a0 0x7f60f81deb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:56278/0 (socket says 172.21.15.100:56278) 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60f659c640 1 -- 172.21.15.100:0/2111724999 learned_addr learned my addr 172.21.15.100:0/2111724999 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:47:06.526 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8125ec0 con 0x7f60f814a700 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f81275e0 con 0x7f60f80b0770 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8126350 con 0x7f60f80ac5a0 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60fccb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60fccb3640 10 monclient(hunting): authenticate will time out at 1546.704336s 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60e6ffd640 10 client.?.objecter ms_handle_connect 0x7f60f80ac5a0 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60e6ffd640 10 client.?.objecter ms_handle_connect 0x7f60f80b0770 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943235228 0 0) 0x7f60f81275e0 con 0x7f60f80b0770 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:06.527 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.509+0000 7f60e6ffd640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:47:06.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d8004eb0 con 0x7f60f80b0770 2024-03-29T11:47:06.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626354563 0 0) 0x7f60f8126350 con 0x7f60f80ac5a0 2024-03-29T11:47:06.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:06.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:06.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:06.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): my global_id is 4700 2024-03-29T11:47:06.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f81275e0 con 0x7f60f80ac5a0 2024-03-29T11:47:06.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 client.?.objecter ms_handle_connect 0x7f60f814a700 2024-03-29T11:47:06.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:06.528 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198391896 0 0) 0x7f60f8125ec0 con 0x7f60f814a700 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f8126350 con 0x7f60f814a700 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2535566226 0 0) 0x7f60d8004eb0 con 0x7f60f80b0770 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60f8125ec0 con 0x7f60f80b0770 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f60ec001520 con 0x7f60f80b0770 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): dump: 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:06.529 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3441369953 0 0) 0x7f60f8125ec0 con 0x7f60f80b0770 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.510+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 >> v1:172.21.15.100:6790/0 conn(0x7f60f80ac5a0 legacy=0x7f60f81deb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 >> v1:172.21.15.100:6789/0 conn(0x7f60f814a700 legacy=0x7f60f81e6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:47:06.530 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 1 monclient: found mon.b 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f81259a0 con 0x7f60f80b0770 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60fccb3640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60fccb3640 10 monclient: _renew_subs 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60fccb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f8125bf0 con 0x7f60f80b0770 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60fccb3640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:06.531 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60fccb3640 10 monclient: _renew_subs 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60fccb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f81df410 con 0x7f60f80b0770 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60fccb3640 1 librados: init done 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60ec001cd0 con 0x7f60f80b0770 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f60ec001180 con 0x7f60f80b0770 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e6ffd640 10 monclient: dump: 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:06.532 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.511+0000 7f60e7fff640 4 set_mon_vals no callback set 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.513+0000 7f60fccb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.513+0000 7f60fccb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.513+0000 7f60fccb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.513+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60dc005080 con 0x7f60f80b0770 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.514+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f60ec019710 con 0x7f60f80b0770 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.515+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1127350977 0 0) 0x7f60ec0835f0 con 0x7f60f80b0770 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.515+0000 7f60f659c640 10 monclient: get_auth_request con 0x7f60d806dcf0 auth_method 0 2024-03-29T11:47:06.533 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.515+0000 7f60e6ffd640 10 client.4525.objecter ms_dispatch 0x7f60f8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:47:06.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.515+0000 7f60e6ffd640 3 client.4525.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:47:06.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.515+0000 7f60e6ffd640 3 client.4525.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:47:06.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.516+0000 7f60e6ffd640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-29T11:47:06.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.516+0000 7f60e6ffd640 10 client.4525.objecter ms_handle_connect 0x7f60d806dcf0 2024-03-29T11:47:06.534 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.521+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60ec04ed00 con 0x7f60f80b0770 2024-03-29T11:47:06.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.521+0000 7f60e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:06.535 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.521+0000 7f60e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:06.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.897+0000 7f60fccb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:06.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.897+0000 7f60fccb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:06.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.897+0000 7f60fccb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-29T11:47:06.903 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.897+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60dc005330 con 0x7f60f80b0770 2024-03-29T11:47:06.904 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:47:06.904 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":11,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:46:28.911584+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:46:28.911584+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:06.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.898+0000 7f60e6ffd640 1 -- 172.21.15.100:0/2111724999 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1916490066) 0x7f60ec050e00 con 0x7f60f80b0770 2024-03-29T11:47:06.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.898+0000 7f60e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:06.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.898+0000 7f60e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:06.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 10 librados: watch_flush enter 2024-03-29T11:47:06.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 10 librados: watch_flush exit 2024-03-29T11:47:06.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-29T11:47:06.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 10 client.4525.objecter successfully canceled tick 2024-03-29T11:47:06.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 >> v1:172.21.15.100:6816/3220289657 conn(0x7f60d806dcf0 legacy=0x7f60d80701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:06.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 10 monclient: shutdown 2024-03-29T11:47:06.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:06.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 >> v1:172.21.15.191:6789/0 conn(0x7f60f80b0770 legacy=0x7f60f8144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:06.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 shutdown_connections 2024-03-29T11:47:06.908 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 >> 172.21.15.100:0/2111724999 conn(0x7f60f806b660 msgr2=0x7f60f80b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:06.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 shutdown_connections 2024-03-29T11:47:06.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.902+0000 7f60fccb3640 1 -- 172.21.15.100:0/2111724999 wait complete. 2024-03-29T11:47:06.909 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:06.903+0000 7f60fccb3640 1 librados: shutdown 2024-03-29T11:47:06.923 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:13.732 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:13.885 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.879+0000 7f8df1341640 10 monclient: get_monmap_and_config 2024-03-29T11:47:13.886 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.879+0000 7f8df1341640 10 monclient: build_initial_monmap 2024-03-29T11:47:13.886 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.879+0000 7f8df1341640 10 monclient: monmap: 2024-03-29T11:47:13.886 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:47:13.886 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:13.886 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:47:13.880188+0000 2024-03-29T11:47:13.887 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:47:13.880188+0000 2024-03-29T11:47:13.887 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:13.887 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:13.887 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:47:13.887 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:47:13.887 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:47:13.888 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:13.888 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.881+0000 7f8df1341640 1 Processor -- start 2024-03-29T11:47:13.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.881+0000 7f8df1341640 1 -- start start 2024-03-29T11:47:13.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.881+0000 7f8df1341640 10 monclient: init 2024-03-29T11:47:13.889 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.882+0000 7f8df1341640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:13.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.882+0000 7f8df1341640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:47:13.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.882+0000 7f8df1341640 10 monclient(hunting): picked mon.noname-a con 0x7f8de4034850 addr v1:172.21.15.100:6789/0 2024-03-29T11:47:13.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.882+0000 7f8df1341640 10 monclient(hunting): picked mon.noname-c con 0x7f8de40c4ec0 addr v1:172.21.15.191:6789/0 2024-03-29T11:47:13.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.882+0000 7f8df1341640 10 monclient(hunting): picked mon.noname-b con 0x7f8de40c7d30 addr v1:172.21.15.100:6790/0 2024-03-29T11:47:13.890 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.882+0000 7f8df1341640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de40c83c0 con 0x7f8de4034850 2024-03-29T11:47:13.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.882+0000 7f8df1341640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de4033280 con 0x7f8de40c4ec0 2024-03-29T11:47:13.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.882+0000 7f8df1341640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de4029100 con 0x7f8de40c7d30 2024-03-29T11:47:13.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.882+0000 7f8df1341640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:13.891 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.882+0000 7f8df1341640 10 monclient(hunting): authenticate will time out at 1554.077507s 2024-03-29T11:47:13.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.886+0000 7f8debfff640 1 --1- >> v1:172.21.15.100:6789/0 conn(0x7f8de4034850 0x7f8de40c47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6789/0 says I am v1:172.21.15.100:37773/0 (socket says 172.21.15.100:37773) 2024-03-29T11:47:13.892 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.886+0000 7f8debfff640 1 -- 172.21.15.100:0/489483416 learned_addr learned my addr 172.21.15.100:0/489483416 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:47:13.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45688929 0 0) 0x7f8de40c83c0 con 0x7f8de4034850 2024-03-29T11:47:13.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:13.893 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:13.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:13.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:47:13.894 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ddc003520 con 0x7f8de4034850 2024-03-29T11:47:13.896 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139862028 0 0) 0x7f8de4029100 con 0x7f8de40c7d30 2024-03-29T11:47:13.896 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:47:13.896 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:13.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:13.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient(hunting): my global_id is 4703 2024-03-29T11:47:13.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de40c83c0 con 0x7f8de40c7d30 2024-03-29T11:47:13.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 <== mon.0 v1:172.21.15.100:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 342865126 0 0) 0x7f8ddc003520 con 0x7f8de4034850 2024-03-29T11:47:13.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 >> v1:172.21.15.100:6790/0 conn(0x7f8de40c7d30 legacy=0x7f8de4029f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:13.897 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 >> v1:172.21.15.191:6789/0 conn(0x7f8de40c4ec0 legacy=0x7f8de4029800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:13.898 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:47:13.898 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 1 monclient: found mon.noname-a 2024-03-29T11:47:13.898 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:13.898 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.100:6789/0 2024-03-29T11:47:13.898 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 --> v1:172.21.15.100:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de40293e0 con 0x7f8de4034850 2024-03-29T11:47:13.898 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient: _finish_auth 0 2024-03-29T11:47:13.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:47:13.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:13.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 <== mon.0 v1:172.21.15.100:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8dd8001690 con 0x7f8de4034850 2024-03-29T11:47:13.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:13.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:47:13.899 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.888+0000 7f8deaffd640 10 monclient: dump: 2024-03-29T11:47:13.900 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:13.900 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:13.900 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:13.900 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:13.900 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:13.900 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:13.901 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:13.901 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:13.901 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:13.901 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:13.901 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.889+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 <== mon.0 v1:172.21.15.100:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dd80018b0 con 0x7f8de4034850 2024-03-29T11:47:13.901 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.889+0000 7f8df1341640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:47:13.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.889+0000 7f8df1341640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:47:13.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.889+0000 7f8deaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:13.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.889+0000 7f8deaffd640 1 -- 172.21.15.100:0/489483416 <== mon.0 v1:172.21.15.100:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8dd8001c40 con 0x7f8de4034850 2024-03-29T11:47:13.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.889+0000 7f8deaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:13.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.889+0000 7f8deaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:47:13.902 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.889+0000 7f8deaffd640 10 monclient: dump: 2024-03-29T11:47:13.903 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:13.903 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:13.903 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:13.903 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:13.903 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:13.903 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:13.904 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:13.904 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:13.904 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:13.904 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:13.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.889+0000 7f8df1341640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:13.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.889+0000 7f8df1341640 4 set_mon_vals no callback set 2024-03-29T11:47:13.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.890+0000 7f8df1341640 10 monclient: shutdown 2024-03-29T11:47:13.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.890+0000 7f8df1341640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:13.904 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.890+0000 7f8df1341640 1 -- 172.21.15.100:0/489483416 >> v1:172.21.15.100:6789/0 conn(0x7f8de4034850 legacy=0x7f8de40c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:13.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.890+0000 7f8df1341640 1 -- 172.21.15.100:0/489483416 shutdown_connections 2024-03-29T11:47:13.905 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.890+0000 7f8df1341640 1 -- 172.21.15.100:0/489483416 >> 172.21.15.100:0/489483416 conn(0x7f8de40058f0 msgr2=0x7f8de4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:13.913 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.891+0000 7f8df1341640 1 -- 172.21.15.100:0/489483416 shutdown_connections 2024-03-29T11:47:13.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.891+0000 7f8df1341640 1 -- 172.21.15.100:0/489483416 wait complete. 2024-03-29T11:47:13.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.892+0000 7f8df1341640 10 monclient: build_initial_monmap 2024-03-29T11:47:13.914 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.892+0000 7f8df1341640 10 monclient: monmap: 2024-03-29T11:47:13.914 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:47:13.914 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:13.914 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:47:13.915 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:47:13.915 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:13.915 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:13.915 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:47:13.915 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:47:13.916 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:47:13.916 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:13.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.893+0000 7f8df1341640 1 librados: starting msgr at 2024-03-29T11:47:13.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.893+0000 7f8df1341640 1 librados: starting objecter 2024-03-29T11:47:13.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.893+0000 7f8df1341640 1 Processor -- start 2024-03-29T11:47:13.916 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.893+0000 7f8df1341640 1 -- start start 2024-03-29T11:47:13.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.893+0000 7f8df1341640 1 librados: setting wanted keys 2024-03-29T11:47:13.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.893+0000 7f8df1341640 1 librados: calling monclient init 2024-03-29T11:47:13.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.893+0000 7f8df1341640 10 monclient: init 2024-03-29T11:47:13.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.894+0000 7f8df1341640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:13.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.894+0000 7f8df1341640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:47:13.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.894+0000 7f8df1341640 10 monclient(hunting): picked mon.noname-b con 0x7f8de4034850 addr v1:172.21.15.191:6789/0 2024-03-29T11:47:13.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8df1341640 10 monclient(hunting): picked mon.noname-c con 0x7f8de40c4ec0 addr v1:172.21.15.100:6790/0 2024-03-29T11:47:13.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8df1341640 10 monclient(hunting): picked mon.noname-a con 0x7f8de40c7d30 addr v1:172.21.15.100:6789/0 2024-03-29T11:47:13.917 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8df1341640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de402e440 con 0x7f8de40c7d30 2024-03-29T11:47:13.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8df1341640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de402e720 con 0x7f8de4034850 2024-03-29T11:47:13.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8df1341640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de40d3f30 con 0x7f8de40c4ec0 2024-03-29T11:47:13.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8df1341640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:13.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8df1341640 10 monclient(hunting): authenticate will time out at 1554.093218s 2024-03-29T11:47:13.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8deb7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7f8de40c4ec0 0x7f8de402c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:35427/0 (socket says 172.21.15.100:35427) 2024-03-29T11:47:13.918 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8deb7fe640 1 -- 172.21.15.100:0/929723625 learned_addr learned my addr 172.21.15.100:0/929723625 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:47:13.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8de8ff9640 10 client.?.objecter ms_handle_connect 0x7f8de40c4ec0 2024-03-29T11:47:13.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8de8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:13.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.898+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014982108 0 0) 0x7f8de40d3f30 con 0x7f8de40c4ec0 2024-03-29T11:47:13.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:47:13.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:13.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:13.919 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 10 monclient(hunting): my global_id is 4706 2024-03-29T11:47:13.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd0004eb0 con 0x7f8de40c4ec0 2024-03-29T11:47:13.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334733095 0 0) 0x7f8dd0004eb0 con 0x7f8de40c4ec0 2024-03-29T11:47:13.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8de40d3f30 con 0x7f8de40c4ec0 2024-03-29T11:47:13.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8de0001a30 con 0x7f8de40c4ec0 2024-03-29T11:47:13.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:13.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:47:13.920 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 10 monclient(hunting): dump: 2024-03-29T11:47:13.920 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:13.920 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:13.920 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:13.921 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:13.921 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:13.921 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:13.921 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:13.921 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:13.921 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:13.921 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:13.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:13.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4152893504 0 0) 0x7f8de40d3f30 con 0x7f8de40c4ec0 2024-03-29T11:47:13.921 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 >> v1:172.21.15.191:6789/0 conn(0x7f8de4034850 legacy=0x7f8de4166d30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:47:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.899+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 >> v1:172.21.15.100:6789/0 conn(0x7f8de40c7d30 legacy=0x7f8de402ce40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:47:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:47:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 1 monclient: found mon.c 2024-03-29T11:47:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:47:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de402c210 con 0x7f8de40c4ec0 2024-03-29T11:47:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:47:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:47:13.922 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:13.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8de0001d30 con 0x7f8de40c4ec0 2024-03-29T11:47:13.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:13.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7f8de0003a60 con 0x7f8de40c4ec0 2024-03-29T11:47:13.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:13.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:13.923 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8de8ff9640 10 monclient: dump: 2024-03-29T11:47:13.923 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8dea7fc640 4 set_mon_vals no callback set 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8df1341640 5 monclient: authenticate success, global_id 4706 2024-03-29T11:47:13.924 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8df1341640 10 monclient: _renew_subs 2024-03-29T11:47:13.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8df1341640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:47:13.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8df1341640 1 -- 172.21.15.100:0/929723625 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de416dc10 con 0x7f8de40c4ec0 2024-03-29T11:47:13.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8df1341640 10 client.4706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:13.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8df1341640 10 monclient: _renew_subs 2024-03-29T11:47:13.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8df1341640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:47:13.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8df1341640 1 -- 172.21.15.100:0/929723625 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de4167620 con 0x7f8de40c4ec0 2024-03-29T11:47:13.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.902+0000 7f8df1341640 1 librados: init done 2024-03-29T11:47:13.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.903+0000 7f8dd67fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:13.925 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.903+0000 7f8dd67fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.903+0000 7f8dd67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:47:13.926 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.903+0000 7f8dd67fc640 1 -- 172.21.15.100:0/929723625 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de402c400 con 0x7f8de40c4ec0 2024-03-29T11:47:13.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.909+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7f8de0003bc0 con 0x7f8de40c4ec0 2024-03-29T11:47:13.928 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.909+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1127350977 0 0) 0x7f8de0084b90 con 0x7f8de40c4ec0 2024-03-29T11:47:13.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.909+0000 7f8de8ff9640 10 client.4706.objecter ms_dispatch 0x7f8dec000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:47:13.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.909+0000 7f8de8ff9640 3 client.4706.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:47:13.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.909+0000 7f8de8ff9640 3 client.4706.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:47:13.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.909+0000 7f8de8ff9640 20 client.4706.objecter dump_active .. 0 homeless 2024-03-29T11:47:13.929 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.910+0000 7f8debfff640 10 monclient: get_auth_request con 0x7f8dd006dc00 auth_method 0 2024-03-29T11:47:13.930 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.910+0000 7f8de8ff9640 10 client.4706.objecter ms_handle_connect 0x7f8dd006dc00 2024-03-29T11:47:13.930 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.915+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8de00502a0 con 0x7f8de40c4ec0 2024-03-29T11:47:13.930 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.915+0000 7f8de8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:13.930 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:13.915+0000 7f8de8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:14.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.244+0000 7f8dd67fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:14.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.244+0000 7f8dd67fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:14.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.244+0000 7f8dd67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:47:14.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.244+0000 7f8dd67fc640 1 -- 172.21.15.100:0/929723625 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8de4029aa0 con 0x7f8de40c4ec0 2024-03-29T11:47:14.252 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.246+0000 7f8de8ff9640 1 -- 172.21.15.100:0/929723625 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1916490066) 0x7f8de00523a0 con 0x7f8de40c4ec0 2024-03-29T11:47:14.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.246+0000 7f8de8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:14.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.246+0000 7f8de8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:14.254 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:47:14.254 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":11,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:46:28.911584+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:46:28.911584+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:14.257 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.249+0000 7f8dd67fc640 10 librados: watch_flush enter 2024-03-29T11:47:14.257 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.249+0000 7f8dd67fc640 10 librados: watch_flush exit 2024-03-29T11:47:14.257 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 20 client.4706.objecter shutdown clearing up homeless session... 2024-03-29T11:47:14.257 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 10 client.4706.objecter successfully canceled tick 2024-03-29T11:47:14.257 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 1 -- 172.21.15.100:0/929723625 >> v1:172.21.15.100:6816/3220289657 conn(0x7f8dd006dc00 legacy=0x7f8dd00700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:14.257 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 10 monclient: shutdown 2024-03-29T11:47:14.258 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:14.258 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 1 -- 172.21.15.100:0/929723625 >> v1:172.21.15.100:6790/0 conn(0x7f8de40c4ec0 legacy=0x7f8de402c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:14.258 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 1 -- 172.21.15.100:0/929723625 shutdown_connections 2024-03-29T11:47:14.258 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 1 -- 172.21.15.100:0/929723625 >> 172.21.15.100:0/929723625 conn(0x7f8de40058f0 msgr2=0x7f8de40c2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:14.258 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 1 -- 172.21.15.100:0/929723625 shutdown_connections 2024-03-29T11:47:14.258 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 1 -- 172.21.15.100:0/929723625 wait complete. 2024-03-29T11:47:14.258 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:14.250+0000 7f8dd67fc640 1 librados: shutdown 2024-03-29T11:47:14.271 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:21.082 DEBUG:teuthology.orchestra.run.smithi100:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:47:21.240 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.227+0000 7fa784b2b640 10 monclient: get_monmap_and_config 2024-03-29T11:47:21.240 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.227+0000 7fa784b2b640 10 monclient: build_initial_monmap 2024-03-29T11:47:21.240 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.227+0000 7fa784b2b640 10 monclient: monmap: 2024-03-29T11:47:21.240 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:47:21.228125+0000 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:47:21.228125+0000 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-b 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.228+0000 7fa784b2b640 1 Processor -- start 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.228+0000 7fa784b2b640 1 -- start start 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.228+0000 7fa784b2b640 10 monclient: init 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.228+0000 7fa784b2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:21.241 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.229+0000 7fa784b2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.229+0000 7fa784b2b640 10 monclient(hunting): picked mon.noname-c con 0x7fa7800ad1a0 addr v1:172.21.15.191:6789/0 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa784b2b640 10 monclient(hunting): picked mon.noname-a con 0x7fa7800ae710 addr v1:172.21.15.100:6789/0 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa784b2b640 10 monclient(hunting): picked mon.noname-b con 0x7fa7801404a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa77ffff640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fa7801404a0 0x7fa78013d1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:42940/0 (socket says 172.21.15.100:42940) 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa77ffff640 1 -- 172.21.15.100:0/1308240239 learned_addr learned my addr 172.21.15.100:0/1308240239 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa784b2b640 1 -- 172.21.15.100:0/1308240239 --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7800aed30 con 0x7fa7800ae710 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa784b2b640 1 -- 172.21.15.100:0/1308240239 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7800afe20 con 0x7fa7800ad1a0 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa784b2b640 1 -- 172.21.15.100:0/1308240239 --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78013d8f0 con 0x7fa7801404a0 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa784b2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa784b2b640 10 monclient(hunting): authenticate will time out at 1561.425668s 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa77e7fc640 1 -- 172.21.15.100:0/1308240239 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082330086 0 0) 0x7fa78013d8f0 con 0x7fa7801404a0 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa77e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:47:21.242 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa77e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa77e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa77e7fc640 10 monclient(hunting): my global_id is 4709 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.230+0000 7fa77e7fc640 1 -- 172.21.15.100:0/1308240239 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa768003520 con 0x7fa7801404a0 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.231+0000 7fa77e7fc640 1 -- 172.21.15.100:0/1308240239 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1082355013 0 0) 0x7fa768003520 con 0x7fa7801404a0 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.231+0000 7fa77e7fc640 1 -- 172.21.15.100:0/1308240239 >> v1:172.21.15.191:6789/0 conn(0x7fa7800ad1a0 legacy=0x7fa78013fd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.232+0000 7fa77e7fc640 1 -- 172.21.15.100:0/1308240239 >> v1:172.21.15.100:6789/0 conn(0x7fa7800ae710 legacy=0x7fa78013cad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.100:6790/0 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 1 -- 172.21.15.100:0/1308240239 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800af1c0 con 0x7fa7801404a0 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:47:21.243 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 1 -- 172.21.15.100:0/1308240239 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa778001a10 con 0x7fa7801404a0 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: dump: 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:21.244 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 1 -- 172.21.15.100:0/1308240239 <== mon.2 v1:172.21.15.100:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa778001be0 con 0x7fa7801404a0 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 1 -- 172.21.15.100:0/1308240239 <== mon.2 v1:172.21.15.100:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa778002030 con 0x7fa7801404a0 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.233+0000 7fa77e7fc640 10 monclient: dump: 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:21.245 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:21.246 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:21.246 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:21.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.234+0000 7fa784b2b640 5 monclient: authenticate success, global_id 4709 2024-03-29T11:47:21.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.234+0000 7fa784b2b640 10 monclient: get_monmap_and_config success 2024-03-29T11:47:21.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.234+0000 7fa784b2b640 4 set_mon_vals no callback set 2024-03-29T11:47:21.246 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.234+0000 7fa784b2b640 10 monclient: shutdown 2024-03-29T11:47:21.247 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.234+0000 7fa784b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:21.247 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.234+0000 7fa784b2b640 1 -- 172.21.15.100:0/1308240239 >> v1:172.21.15.100:6790/0 conn(0x7fa7801404a0 legacy=0x7fa78013d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.235+0000 7fa784b2b640 1 -- 172.21.15.100:0/1308240239 shutdown_connections 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.235+0000 7fa784b2b640 1 -- 172.21.15.100:0/1308240239 >> 172.21.15.100:0/1308240239 conn(0x7fa7800673e0 msgr2=0x7fa7800b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.238+0000 7fa784b2b640 1 -- 172.21.15.100:0/1308240239 shutdown_connections 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.240+0000 7fa784b2b640 1 -- 172.21.15.100:0/1308240239 wait complete. 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.241+0000 7fa784b2b640 10 monclient: build_initial_monmap 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.241+0000 7fa784b2b640 10 monclient: monmap: 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:epoch 0 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:last_changed 0.000000 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:created 0.000000 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 0 (unknown) 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:21.249 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.noname-a 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.noname-c 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.241+0000 7fa784b2b640 1 librados: starting msgr at 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.241+0000 7fa784b2b640 1 librados: starting objecter 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.241+0000 7fa784b2b640 1 Processor -- start 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.242+0000 7fa784b2b640 1 -- start start 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.242+0000 7fa784b2b640 1 librados: setting wanted keys 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.242+0000 7fa784b2b640 1 librados: calling monclient init 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.242+0000 7fa784b2b640 10 monclient: init 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.243+0000 7fa784b2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:47:21.250 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.243+0000 7fa784b2b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:47:21.251 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.243+0000 7fa784b2b640 10 monclient(hunting): picked mon.noname-c con 0x7fa7800ad1a0 addr v1:172.21.15.100:6790/0 2024-03-29T11:47:21.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.244+0000 7fa784b2b640 10 monclient(hunting): picked mon.noname-a con 0x7fa7800ae710 addr v1:172.21.15.100:6789/0 2024-03-29T11:47:21.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.244+0000 7fa784b2b640 10 monclient(hunting): picked mon.noname-b con 0x7fa7801404a0 addr v1:172.21.15.191:6789/0 2024-03-29T11:47:21.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.244+0000 7fa784b2b640 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78006ef50 con 0x7fa7800ae710 2024-03-29T11:47:21.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.244+0000 7fa784b2b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78006f180 con 0x7fa7801404a0 2024-03-29T11:47:21.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.244+0000 7fa784b2b640 1 -- --> v1:172.21.15.100:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7801d2c10 con 0x7fa7800ad1a0 2024-03-29T11:47:21.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.244+0000 7fa784b2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:47:21.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.244+0000 7fa784b2b640 10 monclient(hunting): authenticate will time out at 1561.439815s 2024-03-29T11:47:21.253 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa77f7fe640 1 --1- >> v1:172.21.15.100:6790/0 conn(0x7fa7800ad1a0 0x7fa7801d3130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.100:6790/0 says I am v1:172.21.15.100:38696/0 (socket says 172.21.15.100:38696) 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa77f7fe640 1 -- 172.21.15.100:0/2166521143 learned_addr learned my addr 172.21.15.100:0/2166521143 (peer_addr_for_me v1:172.21.15.100:0/0) 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 10 client.?.objecter ms_handle_connect 0x7fa7800ad1a0 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 <== mon.2 v1:172.21.15.100:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560481105 0 0) 0x7fa7801d2c10 con 0x7fa7800ad1a0 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 10 monclient(hunting): _negotiate 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 10 monclient(hunting): my global_id is 4712 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 --> v1:172.21.15.100:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c004eb0 con 0x7fa7800ad1a0 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 10 client.?.objecter ms_handle_connect 0x7fa7800ae710 2024-03-29T11:47:21.254 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.247+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 <== mon.2 v1:172.21.15.100:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1080114717 0 0) 0x7fa76c004eb0 con 0x7fa7800ad1a0 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.248+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 --> v1:172.21.15.100:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7801d2c10 con 0x7fa7800ad1a0 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.248+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 <== mon.2 v1:172.21.15.100:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa770001870 con 0x7fa7800ad1a0 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.248+0000 7fa767fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.248+0000 7fa767fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.248+0000 7fa767fff640 10 monclient(hunting): dump: 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:21.255 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:21.256 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:21.256 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:21.256 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.248+0000 7fa767fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:47:21.256 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.248+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 <== mon.2 v1:172.21.15.100:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1088220289 0 0) 0x7fa7801d2c10 con 0x7fa7800ad1a0 2024-03-29T11:47:21.256 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.248+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 >> v1:172.21.15.191:6789/0 conn(0x7fa7801404a0 legacy=0x7fa7801de2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.250+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 >> v1:172.21.15.100:6789/0 conn(0x7fa7800ae710 legacy=0x7fa780137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.250+0000 7fa767fff640 10 monclient: _finish_hunting 0 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.250+0000 7fa767fff640 1 monclient: found mon.c 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.250+0000 7fa767fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa767fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 --> v1:172.21.15.100:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7801d3840 con 0x7fa7800ad1a0 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa767fff640 10 monclient: _finish_auth 0 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa767fff640 10 monclient: _check_auth_tickets 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa767fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa784b2b640 5 monclient: authenticate success, global_id 4712 2024-03-29T11:47:21.259 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa784b2b640 10 monclient: _renew_subs 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa784b2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa784b2b640 1 -- 172.21.15.100:0/2166521143 --> v1:172.21.15.100:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7800b2420 con 0x7fa7800ad1a0 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa784b2b640 10 client.4712.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa784b2b640 10 monclient: _renew_subs 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa784b2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa784b2b640 1 -- 172.21.15.100:0/2166521143 --> v1:172.21.15.100:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78006e040 con 0x7fa7800ad1a0 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.251+0000 7fa784b2b640 1 librados: init done 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.252+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 <== mon.2 v1:172.21.15.100:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa770001e00 con 0x7fa7800ad1a0 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.252+0000 7fa767fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.252+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 <== mon.2 v1:172.21.15.100:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4129711964 0 0) 0x7fa770004aa0 con 0x7fa7800ad1a0 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.252+0000 7fa767fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.252+0000 7fa767fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.252+0000 7fa767fff640 10 monclient: dump: 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:epoch 1 2024-03-29T11:47:21.260 INFO:teuthology.misc.health.smithi100.stderr:fsid fd8785f3-b448-4d0b-b3ce-b69d1146f339 2024-03-29T11:47:21.261 INFO:teuthology.misc.health.smithi100.stderr:last_changed 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:21.261 INFO:teuthology.misc.health.smithi100.stderr:created 2024-03-29T11:35:45.837598+0000 2024-03-29T11:47:21.261 INFO:teuthology.misc.health.smithi100.stderr:min_mon_release 19 (squid) 2024-03-29T11:47:21.261 INFO:teuthology.misc.health.smithi100.stderr:election_strategy: 1 2024-03-29T11:47:21.261 INFO:teuthology.misc.health.smithi100.stderr:0: v1:172.21.15.100:6789/0 mon.a 2024-03-29T11:47:21.261 INFO:teuthology.misc.health.smithi100.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-29T11:47:21.261 INFO:teuthology.misc.health.smithi100.stderr:2: v1:172.21.15.100:6790/0 mon.c 2024-03-29T11:47:21.261 INFO:teuthology.misc.health.smithi100.stderr: 2024-03-29T11:47:21.261 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.254+0000 7fa784b2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:21.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.254+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 <== mon.2 v1:172.21.15.100:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2691253245 0 0) 0x7fa770004c00 con 0x7fa7800ad1a0 2024-03-29T11:47:21.262 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.253+0000 7fa77d7fa640 4 set_mon_vals no callback set 2024-03-29T11:47:21.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.254+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 <== mon.2 v1:172.21.15.100:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1127350977 0 0) 0x7fa770084a50 con 0x7fa7800ad1a0 2024-03-29T11:47:21.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.254+0000 7fa767fff640 10 client.4712.objecter ms_dispatch 0x7fa780000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:47:21.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.254+0000 7fa767fff640 3 client.4712.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:47:21.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.254+0000 7fa767fff640 3 client.4712.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:47:21.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.254+0000 7fa767fff640 20 client.4712.objecter dump_active .. 0 homeless 2024-03-29T11:47:21.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.255+0000 7fa784b2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:21.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.255+0000 7fa784b2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:47:21.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.255+0000 7fa784b2b640 1 -- 172.21.15.100:0/2166521143 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7801d2a70 con 0x7fa7800ad1a0 2024-03-29T11:47:21.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.256+0000 7fa77effd640 10 monclient: get_auth_request con 0x7fa76c06dc60 auth_method 0 2024-03-29T11:47:21.263 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.256+0000 7fa767fff640 10 client.4712.objecter ms_handle_connect 0x7fa76c06dc60 2024-03-29T11:47:21.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.265+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 <== mon.2 v1:172.21.15.100:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7801d2a70 con 0x7fa7800ad1a0 2024-03-29T11:47:21.270 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.265+0000 7fa767fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:47:21.271 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.265+0000 7fa767fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:47:21.598 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.593+0000 7fa784b2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:21.598 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.593+0000 7fa784b2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:21.598 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.593+0000 7fa784b2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.100:6790/0 2024-03-29T11:47:21.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.593+0000 7fa784b2b640 1 -- 172.21.15.100:0/2166521143 --> v1:172.21.15.100:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7801d2a70 con 0x7fa7800ad1a0 2024-03-29T11:47:21.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.594+0000 7fa767fff640 1 -- 172.21.15.100:0/2166521143 <== mon.2 v1:172.21.15.100:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1916490066) 0x7fa7801d2a70 con 0x7fa7800ad1a0 2024-03-29T11:47:21.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.594+0000 7fa767fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:47:21.599 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.594+0000 7fa767fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:47:21.599 INFO:teuthology.misc.health.smithi100.stdout: 2024-03-29T11:47:21.599 INFO:teuthology.misc.health.smithi100.stdout:{"epoch":11,"fsid":"fd8785f3-b448-4d0b-b3ce-b69d1146f339","created":"2024-03-29T11:36:23.938571+0000","modified":"2024-03-29T11:46:28.911584+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:46:28.911584+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bafc88bc-7772-46af-81dc-9ab1f21e7bbd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"0a9f0aba-7c68-4b3a-b9cb-edbb25290719","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c2f9c767-038c-4127-b25a-7ed2e956cef1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"75820eae-51d4-4b75-b4be-f45edc929288","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655e63aa-3727-409b-b9e2-652246de4657","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fee1d4d7-1159-4388-ad5f-82e515d74b23","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"559b1250-e1fa-43cf-b727-d47eae0a4a54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23c4000f-1f04-48ef-a47c-9b8d2035383d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:47:21.603 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.597+0000 7fa784b2b640 10 librados: watch_flush enter 2024-03-29T11:47:21.603 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.597+0000 7fa784b2b640 10 librados: watch_flush exit 2024-03-29T11:47:21.603 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.597+0000 7fa784b2b640 20 client.4712.objecter shutdown clearing up homeless session... 2024-03-29T11:47:21.603 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.597+0000 7fa784b2b640 10 client.4712.objecter successfully canceled tick 2024-03-29T11:47:21.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.597+0000 7fa784b2b640 1 -- 172.21.15.100:0/2166521143 >> v1:172.21.15.100:6816/3220289657 conn(0x7fa76c06dc60 legacy=0x7fa76c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:21.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.597+0000 7fa784b2b640 10 monclient: shutdown 2024-03-29T11:47:21.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.597+0000 7fa784b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:47:21.604 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.597+0000 7fa784b2b640 1 -- 172.21.15.100:0/2166521143 >> v1:172.21.15.100:6790/0 conn(0x7fa7800ad1a0 legacy=0x7fa7801d3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:47:21.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.598+0000 7fa784b2b640 1 -- 172.21.15.100:0/2166521143 shutdown_connections 2024-03-29T11:47:21.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.598+0000 7fa784b2b640 1 -- 172.21.15.100:0/2166521143 >> 172.21.15.100:0/2166521143 conn(0x7fa7800673e0 msgr2=0x7fa7800ad9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:47:21.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.598+0000 7fa784b2b640 1 -- 172.21.15.100:0/2166521143 shutdown_connections 2024-03-29T11:47:21.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.599+0000 7fa784b2b640 1 -- 172.21.15.100:0/2166521143 wait complete. 2024-03-29T11:47:21.605 INFO:teuthology.misc.health.smithi100.stderr:2024-03-29T11:47:21.599+0000 7fa784b2b640 1 librados: shutdown 2024-03-29T11:47:21.627 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:47:21.627 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:47:21.628 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:47:21.629 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:47:21.629 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:21.671 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:47:21.672 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:47:21.672 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:21.709 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:47:21.709 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:47:21.709 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:21.783 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:47:21.783 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:47:21.783 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:21.851 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:47:21.851 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:47:21.851 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:21.889 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:47:21.889 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:47:21.889 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:21.964 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:47:21.965 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:47:21.965 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:22.040 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:47:22.040 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:47:22.041 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:22.111 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:47:22.112 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:47:22.112 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:47:22.112 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:22.136 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:47:22.137 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:47:22.137 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:22.180 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:47:22.389 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:47:22.389 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:47:22.389 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:22.402 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:47:22.402 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:47:22.402 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:22.461 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:47:22.461 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:47:22.461 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:47:22.474 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:47:22.474 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:47:22.475 DEBUG:teuthology.orchestra.run.smithi100:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:47:22.714 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:47:22.714 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:47:27.543 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:47:27.543 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:47:27.646 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:47:27.647 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:47:27.712 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:47:27.712 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:47:27.797 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:47:27.798 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:47:30.965 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:47:30.965 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:47:31.070 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:47:31.070 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:47:31.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:47:31.144 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:47:31.202 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:47:31.203 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732/data/mon.a.tgz 2024-03-29T11:47:31.203 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-03-29T11:47:31.221 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.rIoIr2IM5E 2024-03-29T11:47:31.222 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.rIoIr2IM5E 2024-03-29T11:47:31.348 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.rIoIr2IM5E 2024-03-29T11:47:31.437 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.rIoIr2IM5E is 212KB 2024-03-29T11:47:31.538 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.rIoIr2IM5E 2024-03-29T11:47:31.555 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732/data/mon.c.tgz 2024-03-29T11:47:31.555 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-03-29T11:47:31.608 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.kv7Vz8haCw 2024-03-29T11:47:31.609 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.kv7Vz8haCw 2024-03-29T11:47:31.709 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.kv7Vz8haCw 2024-03-29T11:47:31.796 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.kv7Vz8haCw is 212KB 2024-03-29T11:47:31.889 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.kv7Vz8haCw 2024-03-29T11:47:31.904 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732/data/mon.b.tgz 2024-03-29T11:47:31.905 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-03-29T11:47:31.923 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.cnB65UrjSB 2024-03-29T11:47:31.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.cnB65UrjSB 2024-03-29T11:47:32.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.cnB65UrjSB 2024-03-29T11:47:32.135 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.cnB65UrjSB is 212KB 2024-03-29T11:47:32.301 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.cnB65UrjSB 2024-03-29T11:47:32.322 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:47:32.324 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:47:32.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:47:32.417 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:47:32.456 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:47:32.457 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:47:32.463 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732/remote/smithi100/crash 2024-03-29T11:47:32.468 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:47:32.498 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732/remote/smithi191/crash 2024-03-29T11:47:32.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:47:32.527 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:47:32.528 DEBUG:teuthology.orchestra.run.smithi100:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:47:32.544 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:47:32.619 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:47:32.619 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34189.log 2024-03-29T11:47:32.619 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34229.log 2024-03-29T11:47:32.619 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.34189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34469.log 2024-03-29T11:47:32.619 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.34229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34309.log 2024-03-29T11:47:32.619 INFO:teuthology.orchestra.run.smithi191.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34189.log.gz 2024-03-29T11:47:32.619 INFO:teuthology.orchestra.run.smithi191.stderr:gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34229.log.gz -5 2024-03-29T11:47:32.619 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.34469.log: /var/log/ceph/ceph-client.admin.34309.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.audit.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34469.log.gz 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34389.log 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34309.log.gz 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34429.log 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.34389.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.34429.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34389.log.gz 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34429.log.gz 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34269.log 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:47:32.620 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.34269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34349.log 2024-03-29T11:47:32.621 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34269.log.gz 2024-03-29T11:47:32.621 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:47:32.621 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.34349.log: 87.2%/var/log/ceph/ceph-mon.b.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:47:32.621 INFO:teuthology.orchestra.run.smithi191.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34349.log.gz 2024-03-29T11:47:32.621 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:47:32.633 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41731.log 2024-03-29T11:47:32.633 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41848.log 2024-03-29T11:47:32.633 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38862.log 2024-03-29T11:47:32.633 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39253.log 2024-03-29T11:47:32.634 INFO:teuthology.orchestra.run.smithi100.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.41731.log.gz 2024-03-29T11:47:32.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38862.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:47:32.634 INFO:teuthology.orchestra.run.smithi100.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41848.log.gz 2024-03-29T11:47:32.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41887.log 2024-03-29T11:47:32.634 INFO:teuthology.orchestra.run.smithi100.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38862.log.gz 2024-03-29T11:47:32.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41614.log 2024-03-29T11:47:32.634 INFO:teuthology.orchestra.run.smithi100.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39253.log.gz 2024-03-29T11:47:32.634 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40245.log 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39840.log 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41887.log.gz 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38901.log 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41614.log.gz 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39135.log 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40245.log.gz 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38940.log 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39840.log.gz 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39135.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38901.log.gz 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41692.log 2024-03-29T11:47:32.635 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39645.log 2024-03-29T11:47:32.636 INFO:teuthology.orchestra.run.smithi100.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39135.log.gz 2024-03-29T11:47:32.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40754.log 2024-03-29T11:47:32.636 INFO:teuthology.orchestra.run.smithi100.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38940.log.gz 2024-03-29T11:47:32.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41809.log 2024-03-29T11:47:32.636 INFO:teuthology.orchestra.run.smithi100.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41692.log.gz 2024-03-29T11:47:32.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40754.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39645.log.gz 2024-03-29T11:47:32.636 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40871.log 2024-03-29T11:47:32.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40191.log 2024-03-29T11:47:32.636 INFO:teuthology.orchestra.run.smithi100.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2024-03-29T11:47:32.636 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41809.log.gz 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39762.log 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40871.log.gz 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41105.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40191.log.gz 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39370.log 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41066.log 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39370.log: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.39762.log.gz 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41770.log 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38548.log 2024-03-29T11:47:32.637 INFO:teuthology.orchestra.run.smithi100.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39370.log.gz 2024-03-29T11:47:32.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41653.log 2024-03-29T11:47:32.638 INFO:teuthology.orchestra.run.smithi100.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41066.log.gz 2024-03-29T11:47:32.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38548.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41770.log.gz 2024-03-29T11:47:32.638 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41379.log 2024-03-29T11:47:32.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41653.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38548.log.gz 2024-03-29T11:47:32.638 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41340.log 2024-03-29T11:47:32.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39057.log 2024-03-29T11:47:32.638 INFO:teuthology.orchestra.run.smithi100.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41653.log.gz 2024-03-29T11:47:32.638 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38823.log 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41379.log.gz 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39057.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41340.log.gz 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39565.log 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40364.log 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39057.log.gz 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39565.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38823.log.gz 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40364.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39565.log.gz 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40152.log 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38706.log 2024-03-29T11:47:32.639 INFO:teuthology.orchestra.run.smithi100.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40364.log.gz 2024-03-29T11:47:32.640 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39174.log 2024-03-29T11:47:32.640 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38706.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:47:32.640 INFO:teuthology.orchestra.run.smithi100.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40152.log.gz 2024-03-29T11:47:32.640 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38626.log 2024-03-29T11:47:32.640 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40637.log 2024-03-29T11:47:32.640 INFO:teuthology.orchestra.run.smithi100.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38706.log.gz 2024-03-29T11:47:32.640 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38626.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39174.log.gz 2024-03-29T11:47:32.640 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41575.log 2024-03-29T11:47:32.641 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40403.log 2024-03-29T11:47:32.641 INFO:teuthology.orchestra.run.smithi100.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38626.log.gz 2024-03-29T11:47:32.641 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41575.log: gzip -5 85.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40637.log.gz /var/log/ceph/ceph-client.admin.39331.log 2024-03-29T11:47:32.641 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:47:32.641 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40403.log: 86.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.41575.log.gz 2024-03-29T11:47:32.641 INFO:teuthology.orchestra.run.smithi100.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40832.log 2024-03-29T11:47:32.641 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41418.log 2024-03-29T11:47:32.641 INFO:teuthology.orchestra.run.smithi100.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40403.log.gz 2024-03-29T11:47:32.642 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40832.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 85.7% 2024-03-29T11:47:32.642 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39331.log.gz 2024-03-29T11:47:32.642 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41418.log: gzip -5 --verbose -- 86.6% /var/log/ceph/ceph-client.admin.40559.log -- replaced with /var/log/ceph/ceph-client.admin.40832.log.gz 2024-03-29T11:47:32.642 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:47:32.642 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39526.log 86.6% 2024-03-29T11:47:32.642 INFO:teuthology.orchestra.run.smithi100.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41418.log.gz 2024-03-29T11:47:32.642 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40481.log 2024-03-29T11:47:32.642 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41184.log 2024-03-29T11:47:32.642 INFO:teuthology.orchestra.run.smithi100.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40559.log.gz 2024-03-29T11:47:32.643 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40481.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39526.log.gz 2024-03-29T11:47:32.643 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38979.log 2024-03-29T11:47:32.643 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41184.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40481.log.gz 2024-03-29T11:47:32.643 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39879.log 2024-03-29T11:47:32.643 INFO:teuthology.orchestra.run.smithi100.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41184.log.gz 2024-03-29T11:47:32.643 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40074.log 2024-03-29T11:47:32.643 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39487.log 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39879.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38979.log.gz 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40598.log 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39879.log.gz 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39018.log 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40074.log.gz 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38470.log 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39487.log.gz 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40598.log.gz 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39957.log 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39018.log.gz 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39448.log 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40676.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38470.log.gz 2024-03-29T11:47:32.644 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39957.log.gz 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39448.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40442.log 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39448.log.gz 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35031.log 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39292.log 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40442.log.gz 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39801.log 2024-03-29T11:47:32.645 INFO:teuthology.orchestra.run.smithi100.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2024-03-29T11:47:32.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-03-29T11:47:32.646 INFO:teuthology.orchestra.run.smithi100.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35031.log.gz 2024-03-29T11:47:32.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41262.log 2024-03-29T11:47:32.646 INFO:teuthology.orchestra.run.smithi100.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39292.log.gz 2024-03-29T11:47:32.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38665.log 2024-03-29T11:47:32.646 INFO:teuthology.orchestra.run.smithi100.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39801.log.gz 2024-03-29T11:47:32.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2024-03-29T11:47:32.646 INFO:teuthology.orchestra.run.smithi100.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2024-03-29T11:47:32.646 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41027.log 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41262.log.gz 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39604.log 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40988.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38665.log.gz/var/log/ceph/ceph-client.admin.41027.log: 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41536.log 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39604.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28933.log 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41027.log.gz 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39604.log.gz 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40910.log 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:47:32.647 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph.tmp-client.admin.28933.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28933.log.gz 2024-03-29T11:47:32.648 INFO:teuthology.orchestra.run.smithi100.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41536.log.gz 2024-03-29T11:47:32.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41223.log 2024-03-29T11:47:32.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40910.log.gz 2024-03-29T11:47:32.648 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:47:32.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40520.log 2024-03-29T11:47:32.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41457.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41223.log.gz 2024-03-29T11:47:32.648 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38784.log 2024-03-29T11:47:32.648 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40520.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38431.log 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38784.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40520.log.gz 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38745.log 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38431.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38784.log.gz 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39996.log 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38745.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38431.log.gz 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40949.log 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39996.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38745.log.gz 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39996.log.gz 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34740.log 2024-03-29T11:47:32.649 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.0.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40949.log.gz 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40325.log 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.34740.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40325.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34740.log.gz 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40325.log.gz 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39214.log 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40113.log 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.39214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40113.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39214.log.gz 2024-03-29T11:47:32.650 INFO:teuthology.orchestra.run.smithi100.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40113.log.gz 2024-03-29T11:47:32.651 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38509.log 2024-03-29T11:47:32.651 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39723.log 2024-03-29T11:47:32.651 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38509.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2024-03-29T11:47:32.651 INFO:teuthology.orchestra.run.smithi100.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38509.log.gz 2024-03-29T11:47:32.651 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41497.log 2024-03-29T11:47:32.651 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39918.log 2024-03-29T11:47:32.651 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41497.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39723.log.gz 2024-03-29T11:47:32.651 INFO:teuthology.orchestra.run.smithi100.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41497.log.gz 2024-03-29T11:47:32.651 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40035.log 2024-03-29T11:47:32.651 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35134.log 2024-03-29T11:47:32.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.40035.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39918.log.gz 2024-03-29T11:47:32.652 INFO:teuthology.orchestra.run.smithi100.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40035.log.gz 2024-03-29T11:47:32.652 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39684.log 2024-03-29T11:47:32.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.35134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41301.log 2024-03-29T11:47:32.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39684.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35134.log.gz 2024-03-29T11:47:32.652 INFO:teuthology.orchestra.run.smithi100.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39684.log.gz 2024-03-29T11:47:32.652 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38587.log 2024-03-29T11:47:32.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.41301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39409.log 2024-03-29T11:47:32.652 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.38587.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41301.log.gz 2024-03-29T11:47:32.653 INFO:teuthology.orchestra.run.smithi100.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38587.log.gz 2024-03-29T11:47:32.653 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39096.log 2024-03-29T11:47:32.653 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39409.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:47:32.653 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-client.admin.39096.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39409.log.gz 2024-03-29T11:47:32.653 INFO:teuthology.orchestra.run.smithi100.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39096.log.gz 2024-03-29T11:47:32.654 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/ceph-mon.c.log: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:47:44.126 INFO:teuthology.orchestra.run.smithi191.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:47:44.643 INFO:teuthology.orchestra.run.smithi100.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:47:44.748 INFO:teuthology.orchestra.run.smithi100.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:47:44.894 INFO:teuthology.orchestra.run.smithi191.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:47:45.380 INFO:teuthology.orchestra.run.smithi191.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:47:48.809 INFO:teuthology.orchestra.run.smithi100.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:47:48.890 INFO:teuthology.orchestra.run.smithi100.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:47:49.005 INFO:teuthology.orchestra.run.smithi191.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:48:23.233 INFO:teuthology.orchestra.run.smithi100.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:48:29.540 INFO:teuthology.orchestra.run.smithi100.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:48:29.543 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-29T11:48:29.543 INFO:teuthology.orchestra.run.smithi100.stderr:real 0m56.985s 2024-03-29T11:48:29.543 INFO:teuthology.orchestra.run.smithi100.stderr:user 2m38.378s 2024-03-29T11:48:29.543 INFO:teuthology.orchestra.run.smithi100.stderr:sys 0m5.644s 2024-03-29T11:48:42.819 INFO:teuthology.orchestra.run.smithi191.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:48:42.821 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-29T11:48:42.821 INFO:teuthology.orchestra.run.smithi191.stderr:real 1m10.231s 2024-03-29T11:48:42.821 INFO:teuthology.orchestra.run.smithi191.stderr:user 1m58.559s 2024-03-29T11:48:42.822 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m4.429s 2024-03-29T11:48:42.822 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:48:42.822 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732/remote/smithi100/log 2024-03-29T11:48:42.823 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:49:05.622 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732/remote/smithi191/log 2024-03-29T11:49:05.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:49:22.619 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:49:22.812 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ccb81c4201d54ecca736a5992b995ec6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:49:22.816 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:49:22.825 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:49:22.832 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:49:22.832 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:49:22.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:49:22.922 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:49:22.930 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:49:22.930 DEBUG:teuthology.orchestra.run.smithi100:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:49:22.933 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:49:22.946 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:49:22.949 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:49:22.951 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:49:22.951 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-03-29T11:49:22.951 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 +376us[ +372us] +/- 74ms 2024-03-29T11:49:22.952 INFO:teuthology.orchestra.run.smithi100.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -145us[ -149us] +/- 69ms 2024-03-29T11:49:22.952 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -241us[ -244us] +/- 69ms 2024-03-29T11:49:22.952 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:49:22.954 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:49:22.954 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-29T11:49:22.954 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +391us[ +391us] +/- 74ms 2024-03-29T11:49:22.954 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -123us[ -129us] +/- 69ms 2024-03-29T11:49:22.954 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -245us[ -250us] +/- 69ms 2024-03-29T11:49:22.954 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:49:22.955 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:49:22.964 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:49:22.964 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:49:22.972 DEBUG:teuthology.orchestra.run.smithi100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:49:23.062 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:49:23.131 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:49:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:49:23.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:49:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:49:23.193 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:49:23.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:49:23.209 INFO:teuthology.task.internal:Duration was 1347.070993 seconds 2024-03-29T11:49:23.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:49:23.217 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:49:23.217 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:49:23.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:49:23.262 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:49:23.278 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:49:23.694 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:49:23.695 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi100.front.sepia.ceph.com 2024-03-29T11:49:23.695 DEBUG:teuthology.orchestra.run.smithi100:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:49:23.715 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-03-29T11:49:23.715 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:49:23.738 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:49:23.739 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:49:23.757 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:49:23.807 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:49:23.807 DEBUG:teuthology.orchestra.run.smithi100:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:49:23.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:49:23.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:49:23.954 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:49:23.954 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:49:23.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:49:24.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:49:24.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:49:24.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:49:24.047 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = core 2024-03-29T11:49:24.074 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-03-29T11:49:24.096 DEBUG:teuthology.orchestra.run.smithi100:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:49:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:49:24.128 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:49:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:49:24.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:49:24.426 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:49:24.465 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732/remote/smithi100 2024-03-29T11:49:24.465 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:49:24.511 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732/remote/smithi191 2024-03-29T11:49:24.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:49:24.556 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:49:24.557 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:49:24.560 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:49:24.603 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:49:24.615 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:49:24.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:49:24.623 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:49:24.623 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:49:24.626 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:49:24.638 INFO:teuthology.orchestra.run.smithi100.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:49 /home/ubuntu/cephtest 2024-03-29T11:49:24.658 INFO:teuthology.orchestra.run.smithi191.stdout: 265594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:49 /home/ubuntu/cephtest 2024-03-29T11:49:24.659 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:49:24.669 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:49:24.687 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:49:24.706 DEBUG:teuthology.nuke:shortname: smithi100 2024-03-29T11:49:24.706 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:49:24.727 DEBUG:teuthology.nuke:shortname: smithi191 2024-03-29T11:49:24.727 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:49:24.730 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:18:45.407652', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO5bf0bJYXI1z2YAYj2AirhKKyd1vhJ3GYcn3DfjFSYffBbXRw3uQpBRHkGxaMS3Jk2YN+vvtOZrRkqzX6g7kVY='} 2024-03-29T11:49:24.750 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622732', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:18:45.409497', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCXqkS0wvhnvppYL6bO+4o1t1ZrPFTGqFlyuLbR513BpWS3w1HfG9bfqSonOx7xNWAp6+8jBREvVeGStEZIeuuo='} 2024-03-29T11:49:24.779 INFO:teuthology.orchestra.console.smithi100:Power off 2024-03-29T11:49:24.779 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:49:24.804 DEBUG:teuthology.orchestra.console.smithi100:power off output: Chassis Power Control: Down/Off 2024-03-29T11:49:24.804 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:49:24.821 INFO:teuthology.orchestra.console.smithi191:Power off 2024-03-29T11:49:24.821 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:49:24.830 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2024-03-29T11:49:24.843 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2024-03-29T11:49:24.843 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:49:24.866 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-29T11:49:28.831 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:49:28.868 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:49:28.944 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2024-03-29T11:49:28.977 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-29T11:49:32.946 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:49:32.978 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:49:33.056 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2024-03-29T11:49:33.087 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2024-03-29T11:49:33.188 INFO:teuthology.orchestra.console.smithi191:Power off completed 2024-03-29T11:49:37.061 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:49:37.170 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off 2024-03-29T11:49:37.271 INFO:teuthology.orchestra.console.smithi100:Power off completed 2024-03-29T11:49:37.375 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1347.0709929466248 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ccb81c4201d54ecca736a5992b995ec6 status: fail success: false 2024-03-29T11:49:37.375 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:49:37.451 INFO:teuthology.run:FAIL